Video generation method and device, equipment, storage medium and program product
By identifying and segmenting the target image, generating sub-image sequences, and generating videos in sequence, the problem of manual operation of converting pictures into videos in the prior art is solved, and the efficient and convenient video generation process is achieved.
Patent Information
- Application Number
- CN202411718453.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-02-28
Smart Images
Figure CN119545118A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as image processing, video generation, and multimedia, and in particular to a video generation method, a video generation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] Currently, converting images into videos requires the use of dedicated video editing software. Users manually edit images into videos through the video editing software, which requires high software operation capabilities of users and has low efficiency in video generation. Summary of the invention
[0003] The embodiments of the present disclosure provide a video generation method, apparatus, device, storage medium, and program product, which can improve the convenience of converting pictures into videos.
[0004] In a first aspect, an embodiment of the present disclosure proposes a video generation method, comprising: identifying and segmenting objects contained in a target picture to obtain a sub-image sequence, wherein the sub-images in the sub-image sequence contain at least one object in the target picture; generating a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
[0005] In a second aspect, an embodiment of the present disclosure proposes a video generating device, comprising: an image processing module, configured to identify and segment objects contained in a target image to obtain a sub-image sequence, wherein the sub-images in the sub-image sequence contain at least one object in the target image; a video generating module, configured to generate a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generation method described in any implementation manner in the first aspect.
[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the video generation method described in any implementation manner in the first aspect when executed.
[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the video generation method described in any implementation manner in the first aspect.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0011] Figure 1 is an exemplary system architecture diagram to which the present disclosure may be applied;
[0012] Figure 2 A flowchart of a video generation method provided in one embodiment of the present disclosure;
[0013] Figure 3 A flowchart of obtaining a sub-image sequence according to a target image in an embodiment of the present disclosure;
[0014] Figure 4 A flowchart of a video generation method provided by another embodiment of the present disclosure;
[0015] Figure 5 A schematic diagram of a specific application scenario of the video generation method provided by an embodiment of the present disclosure;
[0016] Figure 6 A structural block diagram of a video generation device provided in an embodiment of the present disclosure;
[0017] Figure 7 A schematic structural diagram of an electronic device suitable for executing a video generation method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.
[0019] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0020] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the video generation method, apparatus, device, storage medium, and program product of the present disclosure may be applied.
[0021] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0022] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for realizing information communication between the two, such as image processing applications.
[0023] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when server 105 is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.
[0024] The server 105 can provide various services through various built-in applications. For example, an image processing application that can provide users with image-to-video services can be used. The user can use the terminal devices 101, 102, and 103 to shoot or select a target image and input an image processing request for the target image. The target image and the image processing request are then sent to the server 105 via the network 104. The server 105 runs the image processing application, and identifies and segments the objects contained in the target image according to the image processing request to obtain a sub-image sequence, wherein the sub-images in the sub-image sequence contain at least one object in the target image and the position information of the object in the target image. After obtaining the sub-image sequence of the target image, the server 105 can return the sub-image sequence to the terminal devices 101, 102, and 103 via the network 104. The terminal devices 101, 102, and 103 use the sub-images in the sub-image sequence as video key frames, generate a target video according to the order of the sub-images in the sub-image sequence, and present it to the user.
[0025] It should be noted that the method for converting picture to video service provided in the embodiments of the present disclosure may be executed by a server, may be executed by a terminal device, or may be executed by a server and a terminal device in cooperation with each other.
[0026] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0027] Please refer to Figure 2 , Figure 2 This is a flowchart of a video generation method provided by an embodiment of the present disclosure, wherein process 200 may include the following steps:
[0028] Step 201: Identify and segment objects contained in a target image to obtain a sub-image sequence, wherein a sub-image in the sub-image sequence contains at least one object in the target image;
[0029] This step is intended to be performed by the execution subject of the video generation method (e.g. Figure 1 The server 105 shown in the figure identifies and segments the objects contained in the target image to obtain a sub-image sequence, wherein the sub-images in the sub-image sequence contain at least one object in the target image.
[0030] The target image may be any type of image divided based on various classification criteria, and the embodiments of the present disclosure are not limited thereto. For example, based on the classification criteria of color, the target image may include but is not limited to black and white images and color images; based on the classification criteria of content, the target image may include but is not limited to human images, animal images, landscape images, and architectural images; based on the classification criteria of art form, the target image may include but is not limited to photographic images and painting images.
[0031] The objects included in the target image may include the subject in the image, and may also include the foreground and background in the image, which are not limited in the embodiments of the present disclosure. For example, the subject in the image may include but is not limited to people and animals, and the background in the image may include but is not limited to scenery and buildings; the subject in the image may include but is not limited to rivers and mountains, and the foreground in the image may include but is not limited to ships and vehicles.
[0032] The number of sub-images contained in the sub-image sequence can be determined according to the number of objects contained in the target image and the granularity of identifying and segmenting the target image, which is not limited in the embodiments of the present disclosure. The number of objects in the target image contained in the sub-image can be determined according to the granularity of identifying and segmenting the target image, which is not limited in the embodiments of the present disclosure.
[0033] In the embodiments of the present disclosure, the execution subject may obtain a target image and an image processing request for the target image, and identify and segment the objects contained in the target image according to the image processing request to obtain a sub-image sequence. Figure 1 The terminal devices 101, 102, 103) shown provide a target image and issue an image processing request for the target image.
[0034] The target image may be an image taken or selected by the user. For example, the user may obtain the target image by taking the image with the camera of the terminal device, or the user may select the target image from the photo album of the terminal device, or the user may select the target image from the Internet through the terminal device. The embodiments of the present disclosure do not limit the method for obtaining the target image.
[0035] The image processing request can be a user's demand to convert the target image into a video in various ways. For example, the user can send a voice image processing request through the microphone of the terminal device, the user can also enter a text image processing request through the input box of the terminal device, and the user can also collect an action image processing request through the camera of the terminal device, such as a gesture action.
[0036] In an optional example, the execution subject may identify and segment the objects contained in the target image through a large artificial intelligence model to obtain a sub-image sequence.
[0037] Step 202: Generate a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
[0038] Based on step 201, this step is intended to be performed by the execution subject of the video generation method (for example Figure 1 The terminal devices 101, 102, 103) shown generate a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
[0039] The target video may be a video that displays the objects contained in the sub-images in sequence according to the order of the sub-images in the sub-image sequence, and finally obtains the target image. The execution subject may generate video key frames according to the sub-images in the sub-image sequence. For example, the execution subject may use the first sub-image in the sub-image sequence as the first key frame, generate the second key frame according to the second sub-image and the first key frame, and generate the third key frame according to the third sub-image and the second key frame, until the key frame corresponding to the target image is generated, thereby obtaining the key frame sequence of the target video.
[0040] The execution subject can also generate a video intermediate frame based on two adjacent video key frames, and insert the video intermediate frame between the two adjacent video key frames. The video intermediate frame is used to transition the video picture between the two adjacent video key frames, so that the picture of the target video looks smooth and natural. The embodiment of the present disclosure does not limit the method of generating a video intermediate frame based on two adjacent video key frames, and the number of video intermediate frames generated based on two adjacent video key frames. For example, the video intermediate frame can be generated by linear interpolation or Bezier curve interpolation, and the number of video intermediate frames can be determined according to the smoothness of the video.
[0041] Optionally, after generating the target video, the execution entity may also display the target video to the user through a terminal device.
[0042] The video generation method provided by the embodiments of the present disclosure identifies and segments the objects contained in the target image to obtain a sub-image sequence composed of sub-images containing at least one object in the target image, generates video key frames based on the sub-images, and generates the target video according to the order of the sub-images in the sub-image sequence. The process of converting the image to the video does not require manual operation by the user, which reduces the requirements for the user's software operation ability. The video generation efficiency is high, the convenience of converting the image to the video is improved, and the user experience can be enhanced.
[0043] Please refer to Figure 3 , Figure 3This is a flowchart of obtaining a sub-image sequence according to a target image according to an embodiment of the present disclosure, wherein process 300 may include the following steps:
[0044] Step 301: Identify and segment objects contained in a target image to obtain sub-images;
[0045] This step is intended to be performed by the execution subject of the video generation method (e.g. Figure 1 The server 105 shown in the figure identifies and segments the objects contained in the target image to obtain a sub-image containing at least one object in the target image.
[0046] Among them, when identifying and segmenting the objects contained in the target image, the execution subject can perform image recognition and segmentation of different granularities on the target image according to different video generation requirements. For example, for a target image containing more content, coarse-grained image recognition and segmentation can be performed on the target image, such as semantic segmentation of the target image. For example, for a target image containing less content, fine-grained image recognition and segmentation can be performed on the target image, such as panoramic segmentation of the target image. The execution subject can also perform image recognition and segmentation of different granularities on the same target image according to the different locations of the objects in the target image. For example, fine-grained image recognition and segmentation can be performed for the subject in the target image, and coarse-grained image recognition and segmentation can be performed for the foreground and / or background in the target image, such as instance segmentation of the target image.
[0047] In some optional implementations, step 301 may include: determining the granularity of image recognition and segmentation according to the image processing request; identifying and segmenting the objects contained in the target image according to the granularity of image recognition and segmentation to obtain a sub-image. In addition to coarse granularity and fine granularity, the granularity of image recognition and segmentation may further be refined to include multiple levels. For example, the coarse granularity may include the first to third levels, and the fine granularity may include the fourth to sixth levels. As the granularity level increases, the image is more finely identified and segmented. The granularity of image recognition and segmentation of the target image may be selected by the user when issuing an image processing request through a terminal device. By determining the granularity of image recognition and segmentation, and identifying and segmenting the objects contained in the target image according to the granularity of image recognition and segmentation, the flexibility of video generation may be increased.
[0048] Step 302: determining evaluation parameters of the sub-image according to the type of the object contained in the sub-image and / or the area ratio of the object contained in the sub-image in the target image;
[0049] On the basis of step 301, this step aims to determine the evaluation parameters of the sub-image according to the type of the object contained in the sub-image and / or the area ratio of the object contained in the sub-image in the target image by the above-mentioned execution subject.
[0050] In some optional embodiments, step 302 may include: determining the type of object contained in the sub-image based on the recognition result of the object contained in the target image; determining the proportion of the object contained in the sub-image in the target image based on the ratio of the number of pixels of the object contained in the sub-image to the number of pixels of the target image; determining the evaluation parameters of the sub-image according to preset rules based on the type of object contained in the sub-image and / or the area proportion of the object contained in the sub-image in the target image.
[0051] In an optional example, the execution subject may determine the type of object contained in the sub-image according to the result of the object recognition in step 301 on the target image, and determine the evaluation parameters of the sub-image according to the type of object contained in the sub-image and the preset rules. Optionally, the types of objects contained in the sub-image may include people, animals, plants and inanimate objects, and the preset rules may include: the evaluation parameters of the sub-images containing objects of the types of people, animals, plants and inanimate objects decrease in sequence. Among them, the types of objects contained in the sub-image may not be limited to people, animals, plants and inanimate objects, or people, animals, plants and inanimate objects may be further refined so that they may include multiple types. For example, inanimate objects may include buildings, vehicles, etc., and the embodiments of the present disclosure are not limited to this.
[0052] In another optional example, the execution subject may determine the area ratio of the object contained in the sub-image in the target image according to the number of pixels of the object contained in the sub-image and the number of pixels of the target image, and determine the evaluation parameter of the sub-image according to the area ratio of the object contained in the sub-image in the target image according to the preset rule. Optionally, the preset rule may include: the ratio of the object contained in the sub-image in the target image is positively correlated with the evaluation parameter of the sub-image.
[0053] In another optional example, the execution subject may determine the type of object contained in the sub-image based on the result of identifying the object contained in the target image in step 301. The execution subject may also determine the area ratio of the object contained in the sub-image in the target image based on the number of pixels of the object contained in the sub-image and the target image, and determine the evaluation parameters of the sub-image according to the type of object contained in the sub-image and the area ratio of the object contained in the sub-image in the target image according to the preset rules. This embodiment determines the evaluation parameters of the sub-image based on the type of object contained in the sub-image and the area ratio of the object contained in the sub-image in the target image.
[0054] It should be noted that the evaluation parameters in the embodiments of the present disclosure may be quantitative scores or qualitative grades, and the embodiments of the present disclosure do not limit this. The preset rules for determining the evaluation parameters of the sub-images in the embodiments of the present disclosure may be determined according to the type of the target image, the video generation requirements for the target image, etc., and the embodiments of the present disclosure do not limit this.
[0055] Step 303: Sort the sub-images according to their evaluation parameters to obtain a sub-image sequence.
[0056] On the basis of step 302, this step aims at obtaining a sub-image sequence by sorting the sub-images according to the evaluation parameters of the sub-images by the above-mentioned execution subject.
[0057] In some optional implementations, after obtaining the evaluation parameters of the sub-images, the execution subject may arrange the sub-images in descending order of the evaluation parameters according to the evaluation parameters of the sub-images to obtain a sub-image sequence, so that the sub-images with higher evaluation parameters are arranged at the front of the sub-image sequence, and the sub-images with lower evaluation parameters are arranged at the back of the sub-image sequence, and the sub-images with higher evaluation parameters will be displayed first in the generated target video, and the sub-images with lower evaluation parameters will be displayed later. The manner of sorting the sub-images according to the evaluation parameters of the sub-images may be related to the preset rules for determining the evaluation parameters of the sub-images.
[0058] For example, the evaluation parameter is a quantitative score, the preset rule is that the proportion of the object contained in the sub-image in the target image is positively correlated with the score of the sub-image, and the method of sorting the sub-images according to the scores of the sub-images may be to arrange the sub-images in order from high to low scores. For another example, the evaluation parameter is a quantitative score, the preset rule is that the proportion of the object contained in the sub-image in the target image is negatively correlated with the score of the sub-image, and the method of sorting the sub-images according to the scores of the sub-images may be to arrange the sub-images in order from low to high scores. Among them, the preset rules and the method of sorting the sub-images according to the evaluation parameters of the sub-images need to meet the principle of displaying the subject with a larger area in the target image first in the target video. When the subject with a larger area in the target image is occluded, it is necessary to meet the principle of displaying the unobstructed objects first and then the occluded objects.
[0059] This embodiment determines the evaluation parameters of the sub-images obtained from the target image, and sorts the sub-images according to the evaluation parameters to obtain a sub-image sequence, which can clarify the order in which objects contained in the sub-images are displayed in the target video, making the generated target video more reasonable.
[0060] Please refer to Figure 4 , Figure 4A flowchart of a video generation method provided by another embodiment of the present disclosure, wherein process 400 may include the following steps:
[0061] Step 401: Perform scene recognition and expression recognition on the target image, and determine music matching the target image according to the recognition results;
[0062] This step is intended to be performed by the execution subject of the video generation method (e.g. Figure 1 The server 105 shown in the figure performs scene recognition and expression recognition on the target image, and determines the music matching the target image according to the recognition results.
[0063] Among them, scene recognition may include recognition of weather, for example, it may be recognized whether the weather is sunny, rainy or snowy, etc., and expression recognition may include recognition of whether a person's expression is happy or sad, etc., and the embodiments of the present disclosure are not limited to this. By performing scene recognition and expression recognition on the target image and analyzing the target image from multiple dimensions, the music that matches the atmosphere of the target image can be accurately determined.
[0064] In some optional implementations, step 401 may include: performing scene recognition and expression recognition on the target image, and determining the type of music that matches the target image based on the recognition results; and retrieving music that matches the target image from a preset music resource library based on the music type.
[0065] In an optional example, the execution entity can use an artificial intelligence large model to perform scene recognition and expression recognition on the target image to obtain music that matches the target image.
[0066] Step 402: Identify and segment the objects contained in the target image to obtain a sub-image sequence, wherein a sub-image in the sub-image sequence contains at least one object in the target image;
[0067] Step 403: generating a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video;
[0068] The above steps 402 to 403 are similar to Figure 2 Steps 201 to 202 are the same as shown in the figure. For the same contents, please refer to Figure 2 The corresponding parts of the illustrated implementation manner are not described in detail here.
[0069] Step 404: Set the music as the video soundtrack of the target video.
[0070] Based on step 403, this step is intended to be performed by the above-mentioned execution subject (for example Figure 1The terminal devices 101, 102, and 103 shown in the figure set the music as the video soundtrack of the target video. The implementation method of the present disclosure does not specifically limit the implementation method of setting the music as the video soundtrack of the target video.
[0071] This embodiment performs scene recognition and expression recognition on the target image, determines the music that matches the target image, and combines the music that matches the target image with the target video generated according to the target image, thereby realizing automatic music matching for the video.
[0072] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 5 As shown. User 501 selects a target picture containing four people on the terminal device and issues a picture processing request to convert the target picture into a video (step 502). The cloud server receives the target picture and the picture processing request, and performs scene recognition and expression recognition on the target picture through the artificial intelligence large model according to the picture processing request, obtains music matching the target picture, and recognizes and segments the objects contained in the target picture through the artificial intelligence large model, and obtains a sub-image sequence consisting of five sub-images, wherein the first to fourth sub-images in the sub-image sequence each contain a person in the target picture, and the fifth sub-image contains the background in the target picture (step 503). The terminal device receives the sub-image sequence and the music matching the target picture, generates video key frames according to the sub-images in the sub-image sequence, generates the target video according to the order of the sub-images in the sub-image sequence, and sets the music matching the target picture as the video soundtrack of the target video (step 504). The terminal device displays the final target video to the user (step 505), and the whole process takes less than 1 minute.
[0073] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides a video generation device, such as Figure 6 As shown, the video generating device 600 of this embodiment may include: an image processing module 601 and a video generating module 602 .
[0074] The image processing module 601 is configured to identify and segment the objects contained in the target image to obtain a sub-image sequence, wherein the sub-images in the sub-image sequence contain at least one object in the target image. The video generation module 602 is configured to generate a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
[0075] In this embodiment, the specific processing of the image processing module 601 and the video generation module 602 in the video generation device 600 and the technical effects thereof can be referred to in Figure 2The relevant descriptions of steps 201-202 in the corresponding implementation manner are omitted here.
[0076] In some optional implementations of this embodiment, the image processing module 601 includes:
[0077] A first processing submodule is configured to identify and segment the object contained in the target image to obtain the sub-image;
[0078] An evaluation submodule, configured to determine an evaluation parameter of the subimage according to a type of an object contained in the subimage and / or an area ratio of the object contained in the subimage in the target image;
[0079] The sorting submodule is configured to sort the sub-images according to the evaluation parameters of the sub-images to obtain the sub-image sequence.
[0080] In some optional implementations of this embodiment, the evaluation submodule is further configured as follows:
[0081] Determining the type of the object contained in the sub-image according to the recognition result of the object contained in the target image;
[0082] Determining an area ratio of the object contained in the sub-image in the target image according to a ratio of the number of pixels of the object contained in the sub-image to the number of pixels of the target image;
[0083] The evaluation parameters of the sub-image are determined according to preset rules according to the type of the object included in the sub-image and / or the area ratio of the object included in the sub-image in the target image.
[0084] In some optional implementations of this embodiment, the types of objects contained in the sub-image include: people, animals, plants and inanimate objects;
[0085] The preset rule includes: the evaluation parameters of the sub-images containing objects of the types of people, animals, plants and inanimate objects decrease in sequence.
[0086] In some optional implementations of this embodiment, the preset rule includes: the proportion of the object contained in the sub-image in the target image is positively correlated with the evaluation parameter of the sub-image.
[0087] In some optional implementations of the present embodiment, the sorting submodule is further configured to arrange the sub-images in descending order of the evaluation parameters to obtain the sub-image sequence.
[0088] In some optional implementations of this embodiment, the first processing submodule is further configured as follows:
[0089] Determine the granularity of image recognition and segmentation based on the image processing request;
[0090] The objects contained in the target image are identified and segmented according to the granularity of the image identification and segmentation to obtain the sub-image.
[0091] In some optional implementations of this embodiment, the picture processing module 601 is further configured to perform scene recognition and expression recognition on the target picture, and determine music matching the target picture according to the recognition results;
[0092] The video generation module 602 is further configured to set the music as the video soundtrack of the target video.
[0093] In some optional implementations of this embodiment, the image processing module 601 further includes:
[0094] A second processing submodule is configured to perform scene recognition and expression recognition on the target image, and determine the type of music matching the target image according to the recognition results;
[0095] The retrieval submodule is configured to retrieve music matching the target image in a preset music resource library according to the music type.
[0096] In some optional implementations of the present embodiment, the image processing module 601 is further configured to perform scene recognition and expression recognition on the target image through an artificial intelligence large model to obtain music that matches the target image.
[0097] In some optional implementations of the present embodiment, the image processing module 601 is further configured to identify and segment the objects contained in the target image through an artificial intelligence large model to obtain the sub-image sequence.
[0098] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the video generation method described in any of the above-mentioned embodiments can be implemented when the at least one processor executes.
[0099] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the video generation method described in any of the above embodiments when executed.
[0100] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the video generating method described in any of the above embodiments.
[0101] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0102] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0103] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0104] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).
[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0109] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0110] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0111] According to the video generation method provided by the embodiment of the present disclosure, objects contained in the target image are identified and segmented to obtain a sub-image sequence composed of sub-images containing at least one object in the target image, video key frames are generated according to the sub-images, and the target video is generated according to the order of the sub-images in the sub-image sequence. The process of converting images into videos does not require manual operation by the user, which reduces the requirements for the user's software operation ability. The video generation efficiency is high, the convenience of converting images into videos is improved, and the user experience can be enhanced.
[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A video generation method, comprising: Identifying and segmenting objects contained in a target image to obtain a sub-image sequence, wherein a sub-image in the sub-image sequence contains at least one object in the target image; A target video is generated according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
2. The method according to claim 1, wherein: The identifying and segmenting of the objects contained in the target image to obtain a sub-image sequence includes: Identifying and segmenting the object contained in the target image to obtain the sub-image; Determining an evaluation parameter of the sub-image according to a type of an object included in the sub-image and / or an area ratio of the object included in the sub-image in the target image; The sub-image sequence is obtained by sorting the sub-images according to the evaluation parameters of the sub-images.
3. The method according to claim 2, wherein: Determining the evaluation parameter of the sub-image according to the type of the object contained in the sub-image and / or the area ratio of the object contained in the sub-image in the target image includes: Determining the type of the object contained in the sub-image according to the recognition result of the object contained in the target image; Determining an area ratio of the object contained in the sub-image in the target image according to a ratio of the number of pixels of the object contained in the sub-image to the number of pixels of the target image; The evaluation parameters of the sub-image are determined according to preset rules according to the type of the object included in the sub-image and / or the area ratio of the object included in the sub-image in the target image.
4. The method according to claim 3, wherein: The types of objects contained in the sub-images include: people, animals, plants and inanimate objects; The preset rule includes: the evaluation parameters of the sub-images containing objects of the types of people, animals, plants and inanimate objects decrease in sequence.
5. The method according to claim 3, wherein: The preset rule includes: the area ratio of the object contained in the sub-image in the target image is positively correlated with the evaluation parameter of the sub-image.
6. The method according to claim 5, wherein: The step of sorting the sub-images according to the evaluation parameters of the sub-images to obtain the sub-image sequence includes: The sub-images are arranged in descending order of evaluation parameters to obtain the sub-image sequence.
7. The method according to claim 2, wherein: The identifying and segmenting the object contained in the target image to obtain the sub-image includes: Determine the granularity of image recognition and segmentation based on the image processing request; The objects contained in the target image are identified and segmented according to the granularity of the image identification and segmentation to obtain the sub-image.
8. The method according to claim 1, further comprising: Performing scene recognition and expression recognition on the target image, and determining music matching the target image according to the recognition results; The music is set as the video soundtrack of the target video.
9. The method according to claim 8, wherein: The performing scene recognition and expression recognition on the target image, and determining music matching the target image according to the recognition results, includes: Performing scene recognition and expression recognition on the target image, and determining the type of music matching the target image according to the recognition results; The music matching the target picture is retrieved from a preset music resource library according to the music type.
10. The method according to claim 8, wherein: The performing scene recognition and expression recognition on the target image, and determining music matching the target image according to the recognition results, includes: The target image is subjected to scene recognition and expression recognition through a large artificial intelligence model to obtain music matching the target image.
11. The method according to any one of claims 1 to 10, wherein: The identifying and segmenting of the objects contained in the target image to obtain a sub-image sequence includes: The objects contained in the target image are identified and segmented by a large artificial intelligence model to obtain the sub-image sequence.
12. A video generating device, comprising: An image processing module is configured to identify and segment objects contained in a target image to obtain a sub-image sequence, wherein a sub-image in the sub-image sequence contains at least one object in the target image; The video generation module is configured to generate a target video according to the order of the sub-images in the sub-image sequence, wherein the sub-images are used to generate video key frames in the target video.
13. The device according to claim 12, wherein: The image processing module includes: A first processing submodule is configured to identify and segment the object contained in the target image to obtain the sub-image; An evaluation submodule, configured to determine an evaluation parameter of the subimage according to a type of an object contained in the subimage and / or an area ratio of the object contained in the subimage in the target image; The sorting submodule is configured to sort the sub-images according to the evaluation parameters of the sub-images to obtain the sub-image sequence.
14. The device according to claim 13, wherein: The evaluation submodule is further configured as follows: Determining the type of the object contained in the sub-image according to the recognition result of the object contained in the target image; Determining an area ratio of the object contained in the sub-image in the target image according to a ratio of the number of pixels of the object contained in the sub-image to the number of pixels of the target image; The evaluation parameters of the sub-image are determined according to preset rules according to the type of the object included in the sub-image and / or the area ratio of the object included in the sub-image in the target image.
15. The device according to claim 14, wherein: The types of objects contained in the sub-images include: people, animals, plants and inanimate objects; The preset rule includes: the evaluation parameters of the sub-images containing objects of the types of people, animals, plants and inanimate objects decrease in sequence.
16. The device according to claim 14, wherein: The preset rule includes: the proportion of the object contained in the sub-image in the target image is positively correlated with the evaluation parameter of the sub-image.
17. The device according to claim 16, wherein: The sorting submodule is further configured to arrange the sub-images in descending order of the evaluation parameters to obtain the sub-image sequence.
18. The device according to claim 13, wherein: The first processing submodule is further configured to: Determine the granularity of image recognition and segmentation based on the image processing request; The objects contained in the target image are identified and segmented according to the granularity of the image identification and segmentation to obtain the sub-image.
19. The device according to claim 13, wherein: The image processing module is further configured to perform scene recognition and expression recognition on the target image, and determine music matching the target image according to the recognition results; The video generation module is also configured to set the music as the video soundtrack of the target video.
20. The device according to claim 19, wherein The image processing module further includes: A second processing submodule is configured to perform scene recognition and expression recognition on the target image, and determine the type of music matching the target image according to the recognition results; The retrieval submodule is configured to retrieve music matching the target image in a preset music resource library according to the music type.
21. The device according to claim 20, wherein: The image processing module is also configured to perform scene recognition and expression recognition on the target image through an artificial intelligence large model to obtain music that matches the target image.
22. According to the device according to any one of claims 12-21, the image processing module is also configured to identify and segment the objects contained in the target image through an artificial intelligence large model to obtain the sub-image sequence.
23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of claims 1 to 11.
24. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the video generation method according to any one of claims 1 to 11.
25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 11 are implemented.