Image generation method and device

Abstract

The present application discloses an image generation method and a device thereof. The invention belongs to the field of image processing technology. An embodiment of the method comprises: receiving a first input of a user to a dynamic photo, the dynamic photo comprising multiple frames of images; in response to the first input, fusing the multiple frames of images to obtain a first image; inputting the first image to an image expansion model to obtain a second image output by the image expansion model, the second image comprising the image content of the first image and a first extended element, the first extended element being generated by the image expansion model based on the image content of the first image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and specifically to an image generation method and device thereof. Background Art

[0002] Image expansion is an image processing method that expands or enlarges the original image. By expanding the image, users can generate an image with a larger field of view based on the existing image.

[0003] In the prior art, the image to be expanded is usually input into an artificial intelligence model with an image expansion function to obtain an expanded image. However, the image to be expanded is often a cropped image with little reference information, resulting in low accuracy of the image expansion model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image generation method and device thereof, which can improve the accuracy of image expansion by an image expansion model.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising: receiving a first input of a dynamic photo by a user, the dynamic photo comprising multiple frames of images; in response to the first input, fusing the multiple frames of images to obtain a first image; inputting the first image into an image expansion model to obtain a second image output by the image expansion model, the second image comprising the image content of the first image and a first extension element, the first extension element being generated by the image expansion model based on the image content of the first image.

[0006] In a second aspect, an embodiment of the present application provides an image generating device, which includes: a receiving unit, used to receive a first input of a dynamic photo by a user, wherein the dynamic photo includes multiple frames of images; a fusion unit, used to fuse the multiple frames of images in response to the first input to obtain a first image; and an expansion unit, used to input the first image into an expansion model to obtain a second image output by the expansion model, wherein the second image includes the image content of the first image and a first extension element, and the first extension element is generated by the expansion model based on the image content of the first image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, when a user's first input to a dynamic photo is received, multiple frames of images in the dynamic photo are first fused to obtain a first image, and then the first image is input into an image expansion model to obtain a second image output by the above-mentioned image expansion model, wherein the second image includes the image content of the first image and a first extension element, and the first extension element is generated by the image expansion model based on the image content of the first image. Since multiple frames of images in the dynamic photo are fused before the image expansion is performed by the image expansion model, the information in the dynamic photo can be fully utilized to achieve a preliminary expansion of the image. Using the image expansion model to expand the image on this basis can provide the image expansion model with more reference information, while reducing the image area required to be expanded by the image expansion model, thereby improving the accuracy of the image expansion by the image expansion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of an image generation method provided in an embodiment of the present application;

[0013] Figure 2A is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0014] Figure 2B is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0015] Figure 3A is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0016] Figure 3B is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0017] Figure 3C is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0018] Figure 3D is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0019] Figure 4Ais a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0020] Figure 4B is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0022] Fig. 6A is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0023] Figure 6B is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0024] Figure 6C is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] The image generation method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] Please refer to Figure 1 , which shows one of the flow charts of the image generation method provided in the embodiment of the present application. The image generation method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptops, and wearable devices.

[0032] The process of the image generation method provided in the embodiment of the present application includes the following steps:

[0033] Step 101, receiving a first input of a dynamic photo from a user, where the dynamic photo includes multiple frames of images.

[0034] In this embodiment, a live photo is a photo with dynamic effects, which can also be called a live photo. During the live photo shooting process, the camera can not only record the static photo at the shooting moment, but also capture the picture and sound within a few seconds before and after the shooting moment, making the photo more vivid and real. Therefore, a live photo can include multiple frames of images.

[0035] In this embodiment, the first input can be used to trigger the expansion of the dynamic photo. The first input can be a touch input, or a voice command, or a specific gesture input by the user, or other feasible input, which can be determined according to actual usage requirements, and is not limited in the embodiments of the present application. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture. The click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0036] As an example, see Figure 2A , users can start the dynamic photo shooting function in the camera interface and use it to shoot dynamic images. After taking a photo, if you are not satisfied with the composition of the photo, you can trigger the page jump by clicking a specific control or inputting a voice command, so that the displayed interface jumps from the photo preview interface to the album editing interface. The editing interface can be seen in Figure 2B The user can trigger the expansion of the dynamic photo by clicking the “AI expansion” control shown by label 201.

[0037] Step 102: In response to a first input, multiple frames of images are fused to obtain a first image.

[0038] In this embodiment, in response to the first input, the multiple frames of images may be fused so that the fused first image includes the content of each frame of the multiple frames of images.

[0039] Specifically, one of the above-mentioned multiple frames of images can be first selected as the first target image, and the adjacent image of the first target image can be selected as the second target image. The first target image includes an area A that overlaps with the second target image and an area B that does not exist in the second target image. The second target image includes an area A that overlaps with the first target image and an area C that does not exist in the first target image. Then, the image composed of the above-mentioned area A, area B and area C can be used as the image after the first target image and the second target image are fused, and after the first target image and the second target image are deleted, the fused image is used as the first target image. Then, the above process can be continued until the above-mentioned multiple frames of images are fused into one image. The image finally obtained is the first image.

[0040] Step 103, input the first image into the image expansion model to obtain the second image output by the image expansion model.

[0041] In this embodiment, after obtaining the first image, the first image can be input into a pre-trained image expansion model to obtain a second image output by the image expansion model, wherein the second image intentionally includes the image content of the first image and the first expansion element, and the first expansion element can be generated by the image expansion model based on the image content of the first image.

[0042] In practice, the above-mentioned image expansion model can be a neural network model with an image expansion function. It can expand the input first image based on the image content of the first image, fill its edge area with content related to the first image, and obtain a second image with a larger size. The above-mentioned content related to the first image is the first expanded content. As an example, the image content of the first image is a local area of ​​a badminton court and the players in the local area. The image expansion model can expand the image based on the badminton court displayed in the first image to obtain a second image containing the complete court area.

[0043] The method provided by the above-mentioned embodiment of the present application, upon receiving the first input of the user to the dynamic photo, first fuses the multiple frames of images in the dynamic photo to obtain the first image, and then inputs the first image into the image expansion model, thereby obtaining the second image output by the above-mentioned image expansion model, wherein the second image includes the image content of the first image and the first extension element, and the first extension element is generated by the image expansion model based on the image content of the first image. Since the multiple frames of images in the dynamic photo are fused before the image expansion is performed by the image expansion model, the information in the dynamic photo can be fully utilized to achieve the preliminary expansion of the image. On this basis, using the image expansion model to expand the image can provide the image expansion model with more reference information, while reducing the image area required to be expanded by the image expansion model, thereby improving the accuracy of the image expansion by the image expansion model.

[0044] In some optional embodiments, the above step 102 may further include the following sub-steps:

[0045] Sub-step 1021, in response to the first input, determining a subject in the dynamic photo.

[0046] Here, the subject in the dynamic photo may be one or more photographed objects in the dynamic photo. In practice, the subject in the dynamic photo may be automatically selected, or the subject in the dynamic photo may be selected by receiving user input.

[0047] In some optional implementations, in response to the first input, all the photographed objects involved in the multiple frames of images included in the dynamic photo can be compared, and the N photographed objects that appear the longest are taken as the main body. Where N is a positive integer. As an example, in response to the first input, the following may be displayed: Figure 3A The subject analysis interface shown in . At this time, the system can extract the above multiple frames from the dynamic photo, and automatically analyze the above multiple frames to determine the three subjects that appear the longest as the subjects. Assuming N = 1, since the little girl in the dynamic photo appears the longest, the little girl can be regarded as the subject, and the following is presented: Figure 3B In the subject analysis result page shown, the area where the subject is located is displayed in the original style, as shown by reference numeral 301. The remaining areas can be displayed with reduced brightness.

[0048] By automatically analyzing the subject in the dynamic photo, the user can save the process of manually selecting the subject, thereby reducing the complexity of user operation and improving the convenience of subject determination and image fusion efficiency.

[0049] In some optional implementations, if the automatic analysis does not yield a subject that meets the user's expectations, the user can manually select the subject. Figure 3B, the user can click the "cancel automatically analyzed subject" control as shown in the label 302. At this time, the dynamic photo can be displayed, see Figure 3C When the user long presses the little girl in the dynamic photo as shown in the label 303, the little girl in the dynamic photo can be used as the subject. Further, the dynamic photo can be separated into layers so that the area where the subject is located is displayed in the original style, and the remaining area can be displayed with reduced brightness, see Figure 3D shown.

[0050] By providing the user with the function of manually selecting a subject, even if the automatic analysis does not obtain a subject that meets the user's expectations, the subject selection can still be successfully performed, thereby improving the flexibility of subject selection.

[0051] Sub-step 1022: Based on the subject, multiple frames of images are fused to obtain a first image.

[0052] In practice, the subjects in the multiple frames of images may be aligned first, and then the method in the above embodiment may be used to fuse the multiple frames of images after the subjects have been aligned to obtain a first image. By aligning the subjects in the multiple frames of images, the fusion result of the multiple frames of images may be made clearer, thereby improving the quality of the first image.

[0053] In some optional embodiments, the above sub-step 1022 may further include:

[0054] The first step is to display a setting interface, which includes a canvas, and the canvas includes the above-mentioned subject. As an example, the setting interface is shown in FIG4 , and the setting interface may include a canvas as shown by reference numeral 401 and a subject as shown by reference numeral 402 .

[0055] The second step is to receive the second input from the user to the setting interface. Here, the second input can be used to set at least one of the following parameters: the position of the subject in the canvas, the size of the subject, and the size of the canvas. The second input can be a touch input, or a specific gesture input by the user, or other feasible input, which can be determined according to actual usage requirements and is not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application can be any one or more of a sliding gesture, a dragging gesture, an area change gesture, a double-press gesture, and a double-branch zoom gesture.

[0056] The third step is to set at least one of the following parameters in response to the second input: the position of the subject in the canvas, the size of the subject, and the size of the canvas. As an example, when the user drags the subject in the canvas, the position of the subject in the canvas can be changed to achieve the setting of the subject's position. When the user pinches to zoom the subject, the size of the subject can be changed to achieve the setting of the subject's size. When the user stretches the canvas around, the size of the canvas can be changed to achieve the setting of the canvas size. It should be noted that the size of the canvas is the size of the second image finally generated. The position of the subject in the canvas is the position of the subject in the second image finally generated. The size of the subject is the size of the subject in the second image finally generated.

[0057] The fourth step is to fuse the multiple frames of images based on the subject and the parameters to obtain the first image. Here, the multiple frames of images can be first scaled according to the subject selected by the user and the size of the subject set, so that the size of the subject in the scaled multi-frame images is equal to the set size. Then, according to the position of the user in the canvas, the subjects in the scaled multi-frame images are aligned. After that, the method in the above embodiment is used to fuse the multiple frames of images after the subject is aligned, and the background data already contained in the dynamic photo is completed to obtain the first image. Here, the size of the first image can be equal to the set size of the canvas, and there can be an area in the first image that has not been expanded, and the content of this area can be generated through the expansion model in the subsequent steps.

[0058] As an example, see Figure 5 , the original image is shown as 501. After the user sets the parameters, the electronic device can first compare the multiple frames in the dynamic photo, and fuse the multiple frames based on the subject and the set parameters to obtain the first image as shown in 502. In the first image, the area where the image content exists is shown as 503. After the first image is input into the image expansion model, the second image as shown in 504 output by the image expansion model can be obtained. The size of the second image is equal to the size of the canvas set by the user.

[0059] Because multiple frames of images in the dynamic photo are fused before the image is expanded by the image expansion model, the information in the dynamic photo can be fully utilized to achieve the initial expansion of the image. On this basis, using the image expansion model to expand the image can provide the image expansion model with more reference information, while reducing the image area that the image expansion model needs to expand, thereby improving the accuracy of the image expansion model. In addition, by setting the position of the subject in the canvas, the size of the subject, and the size of the canvas, the generated second image can better meet the personalized needs of users.

[0060] In some optional embodiments, the above step 103 may further include the following sub-steps:

[0061] Sub-step 1031, obtaining audio data corresponding to the dynamic photo.

[0062] In practice, during the dynamic photo shooting process, the audio data collected during the shooting process can be stored synchronously. Here, the audio data corresponding to the dynamic photo can be directly obtained locally.

[0063] Sub-step 1032, determining a second extension element based on the audio data.

[0064] The second extended element may include but is not limited to at least one of the following: animal elements, landscape elements, and humanistic elements. As an example, animal elements may include but are not limited to at least one of the following: cats, dogs, birds, etc. Landscape elements may include but are not limited to at least one of the following: rivers, seas, etc. Humanistic elements may include but are not limited to at least one of the following: vendors hawking, children playing, etc.

[0065] In practice, audio recognition can be performed through artificial intelligence algorithms to determine the sound source of the sound in the audio data. The sound source is used as the second extended element. For example, if the audio data contains the sound of birds singing, the bird can be used as the second extended element. For another example, if the audio data contains the sound of a car passing by, the car can be used as the second extended element.

[0066] Sub-step 1033: Generate prompt information based on the second extension element.

[0067] Here, the text description of the second extension element can be filled with a preset template to obtain a prompt message. For example, the preset template can be "You are a large model with an expansion function. Please use XX as an extension element to expand the following image and generate a new image including the above extension element." If the extension element is a river, the prompt message can be "You are a large model with an expansion function. Please use the river as an extension element to expand the following image and generate a new image including the above extension element."

[0068] Sub-step 1034, inputting the first image and the prompt information into the image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image, the first extended element, and the second extended element.

[0069] Here, the first image and the prompt information can be input into the image expansion model, and the image expansion model can understand the semantics of the prompt information, thereby expanding the first image according to the requirements in the prompt information, and finally outputting the second image. The second image includes not only the image content in the first image, but also the area outside the image content of the first image, and also includes the first expansion element and the second expansion element.

[0070] By introducing the audio information corresponding to the dynamic image, not only can the area to be expanded of the expansion model be reduced, but the information source input is further increased, which can provide a clear generation direction for the expansion model and avoid generating irrelevant content during the expansion process, thereby further improving the accuracy of the expansion model.

[0071] In some optional embodiments, the above sub-step 1032 may further include:

[0072] The first step is to determine at least one candidate element based on the audio data. In practice, you can first use an artificial intelligence algorithm to perform audio recognition, determine the various sound-making objects in the audio data, and use each sound-making object as a candidate element. Then, you can sort the candidate elements according to the duration of the sound of different sound-making objects. During this process, you can display the audio analysis interface, see Fig. 6A shown.

[0073] The second step is to display at least one option corresponding to at least one candidate element. Here, at least one option corresponding to the at least one candidate element can be displayed one by one according to the sorting result. Figure 6B As shown, according to the duration of the sound, the candidate elements are river, dog, and bird in order. The display interface may include an option 601 corresponding to the river, an option 602 corresponding to the dog, and an option 603 corresponding to the bird.

[0074] The third step is to receive a third input from the user for a target option in the at least one option. Here, the target option can be any one of the at least one option. The third input can be used to select a second expansion element in the at least one candidate element.

[0075] Step 4: In response to the third input, determine a second expansion element in at least one candidate element. For example, if the user clicks option 602, dog can be used as the second expansion element.

[0076] This embodiment provides a function for the user to select a second extension element, so that the expansion model can perform expansion processing in a targeted manner, further improving the accuracy of the expansion model.

[0077] In some optional embodiments, the above sub-step 1033 may further include:

[0078] In the first step, the second control is displayed. The second control can be used to provide a text edit box. For example, see Figure 6C As shown in the reference numeral 604.

[0079] The second step is to receive the description information of the second extension element input by the user in the second control. Here, the description information can be used to describe the second extension element. The description information can be edited and input by the user according to memory. For example, if the second extension element is a dog, the description information can include but is not limited to "corgi", "holding a bone", etc.

[0080] The third step is to generate prompt information based on the second extended element and the description information. Here, the text and description information corresponding to the second extended element can be filled into the preset template to obtain the prompt information. For example, the preset template can be "You are a large model with an expansion function. Please use XX as the extended element and XXX as the feature of the extended element to expand the following image and generate a new image including the above extended element." If the extended element is a dog and the description information is "Corgi, holding a bone", the prompt information can be "You are a large model with an expansion function. Please use the dog as the extended element and 'Corgi, holding a bone' as the feature of the extended element to expand the following image and generate a new image including the above extended element."

[0081] By providing an input function for descriptive information of the second extended element, the information source input can be further increased, thereby providing the expansion model with a richer and more accurate generation direction, so that the expansion model can restore the user's memory picture, further improving the accuracy of the expansion model.

[0082] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0083] like Figure 7 As shown, the image generating device 700 of this embodiment includes: a receiving unit 701, used to receive a first input of a dynamic photo by a user, wherein the dynamic photo includes multiple frames of images; a fusion unit 702, used to fuse the multiple frames of images in response to the first input to obtain a first image; an image expansion unit 703, used to input the first image into an image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image and a first extended element, and the first extended element is generated by the image expansion model based on the image content of the first image.

[0084] In some optional implementations of this embodiment, the fusion unit 702 is further used to: determine the subject in the dynamic photo in response to the first input; and fuse the multiple frames of images based on the subject to obtain a first image. By fusing the multiple frames of images based on the subject, the fusion result can be made clearer, thereby improving the quality of the first image.

[0085] In some optional implementations of this embodiment, the fusion unit 702 is further used to: display a setting interface, the setting interface includes a canvas, and the canvas includes the subject; receive a second input from the user to the setting interface; in response to the second input, set at least one of the following parameters: the position of the subject in the canvas, the size of the subject, and the size of the canvas; based on the subject and the parameters, fuse the multiple frames of images to obtain a first image. Since the multiple frames of images in the dynamic photo are fused before the image is expanded by the image expansion model, the information in the dynamic photo can be fully utilized to achieve the initial expansion of the image. On this basis, using the image expansion model to expand the image can provide more reference information to the image expansion model, while reducing the image area required to be expanded by the image expansion model, thereby improving the accuracy of the image expansion by the image expansion model. In addition, by setting the position of the subject in the canvas, the size of the subject, and the size of the canvas, the generated second image can better meet the personalized needs of the user.

[0086] In some optional implementations of this embodiment, the image expansion unit 703 is also used to: obtain audio data corresponding to the dynamic photo; determine the second extended element based on the audio data; generate prompt information based on the second extended element; input the first image and the prompt information into the image expansion model to obtain the second image output by the image expansion model, wherein the second image includes the image content of the first image, the first extended element, and the second extended element. By introducing the audio information corresponding to the dynamic image, not only can the area to be expanded in the image expansion model be reduced, but also the information source input is further increased, and a clear generation direction can be provided for the image expansion model to avoid generating irrelevant content during the image expansion process, thereby further improving the accuracy of the image expansion model.

[0087] In some optional implementations of this embodiment, the image expansion unit 703 is further used to: determine at least one candidate element based on the audio data; display at least one option corresponding to the at least one candidate element; receive a third input from the user for a target option in the at least one option; and determine a second expansion element in the at least one candidate element in response to the third input. Thus, a function of allowing the user to select a second expansion element is provided, so that the image expansion model can perform targeted image expansion processing, further improving the accuracy of the image expansion model.

[0088] In some optional implementations of this embodiment, the image expansion unit 703 is further used to: display a second control; receive description information of the second extended element input by the user in the second control; and generate prompt information based on the second extended element and the description information. By providing an input function for the description information of the second extended element, the information source input can be further increased, thereby providing a richer and more accurate generation direction for the image expansion model, so that the image expansion model can restore the user's memory picture, further improving the accuracy of the image expansion model.

[0089] The device provided by the above-mentioned embodiment of the present application, upon receiving the first input of the user to the dynamic photo, first fuses the multiple frames of images in the dynamic photo to obtain the first image, and then inputs the first image into the image expansion model to obtain the second image output by the image expansion model, wherein the second image includes the image content of the first image and the first extension element, and the first extension element is generated by the image expansion model based on the image content of the first image. Since the multiple frames of images in the dynamic photo are fused before the image expansion is performed by the image expansion model, the information in the dynamic photo can be fully utilized to achieve the preliminary expansion of the image. On this basis, using the image expansion model to expand the image can provide the image expansion model with more reference information, while reducing the image area area that the image expansion model needs to expand, thereby improving the accuracy of the image expansion by the image expansion model.

[0090] The image generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0091] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0092] The image generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0093] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0094] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0095] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0096] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0097] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0098] Among them, the processor 910 is used to receive a first input of a user to a dynamic photo through the user input unit 907, and the dynamic photo includes multiple frames of images; in response to the first input, the multiple frames of images are fused to obtain a first image; the first image is input to the expansion model to obtain a second image output by the expansion model, and the second image includes the image content of the first image and a first extension element, and the first extension element is generated by the expansion model based on the image content of the first image.

[0099] Because multiple frames of images in the dynamic photo are fused before the image expansion model is used to expand the image, the information in the dynamic photo can be fully utilized to achieve the initial expansion of the image. On this basis, using the image expansion model to expand the image can provide the image expansion model with more reference information and reduce the image area that the image expansion model needs to expand, thereby improving the accuracy of the image expansion model.

[0100] In some optional implementations of this embodiment, the processor 910 is further configured to determine a subject in the dynamic photo in response to the first input; and fuse the multiple frames of images based on the subject to obtain a first image. By fusing the multiple frames of images based on the subject, the fusion result can be made clearer, thereby improving the quality of the first image.

[0101] In some optional implementations of this embodiment, the processor 910 is further used to display a setting interface through the display unit 906, wherein the setting interface includes a canvas, and the canvas includes the subject; receive a second input of the user to the setting interface through the user input unit 907; in response to the second input, set at least one of the following parameters: the position of the subject in the canvas, the size of the subject, and the size of the canvas; based on the subject and the parameters, fuse the multiple frames of images to obtain a first image. Since the multiple frames of images in the dynamic photo are fused before the image is expanded by the image expansion model, the information in the dynamic photo can be fully utilized to achieve the preliminary expansion of the image. On this basis, using the image expansion model to expand the image can provide more reference information to the image expansion model, while reducing the image area required to be expanded by the image expansion model, thereby improving the accuracy of the image expansion of the image expansion model. In addition, by setting the position of the subject in the canvas, the size of the subject, and the size of the canvas, the generated second image can better meet the personalized needs of the user.

[0102] In some optional implementations of this embodiment, the processor 910 is further used to obtain audio data corresponding to the dynamic photo; determine the second extended element based on the audio data; generate prompt information based on the second extended element; input the first image and the prompt information into the image expansion model to obtain the second image output by the image expansion model, wherein the second image includes the image content of the first image, the first extended element, and the second extended element. By introducing the audio information corresponding to the dynamic image, not only can the area to be expanded in the image expansion model be reduced, but also the information source input is further increased, and a clear generation direction can be provided for the image expansion model to avoid generating irrelevant content during the image expansion process, thereby further improving the accuracy of the image expansion model.

[0103] In some optional implementations of this embodiment, the processor 910 is further configured to determine at least one candidate element based on the audio data; display at least one option corresponding to the at least one candidate element through the display unit 906; receive a third input of a user to a target option in the at least one option through the user input unit 907; and determine a second expansion element in the at least one candidate element in response to the third input. Thus, a function of allowing the user to select the second expansion element is provided, so that the expansion model can perform expansion processing in a targeted manner, further improving the accuracy of the expansion model in expanding the image.

[0104] In some optional implementations of this embodiment, the processor 910 is further configured to display a second control through the display unit 906; receive description information of the second extended element input by the user in the second control; and generate prompt information based on the second extended element and the description information. By providing an input function for the description information of the second extended element, the information source input can be further increased, thereby providing a richer and more accurate generation direction for the expansion model, so that the expansion model can restore the user's memory picture, further improving the accuracy of the expansion model.

[0105] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0106] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0107] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0108] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0110] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0111] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0112] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0113] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0115] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: receiving a first input of a dynamic photo by a user, wherein the dynamic photo includes a plurality of frames of images; In response to the first input, fusing the multiple frames of images to obtain a first image; The first image is input into an image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image and a first extension element, and the first extension element is generated by the image expansion model based on the image content of the first image.

2. The method according to claim 1, characterized in that: In response to the first input, fusing the multiple frames of images to obtain a first image includes: In response to the first input, determining a subject in the dynamic photo; Based on the subject, the multiple frames of images are fused to obtain a first image.

3. The method according to claim 2, characterized in that The step of fusing the multiple frames of images based on the subject to obtain a first image includes: Displaying a setting interface, wherein the setting interface includes a canvas, and the canvas includes the subject; receiving a second input from a user on the setting interface; In response to the second input, obtaining at least one of the following parameters: a position of the subject in the canvas, a size of the subject, and a size of the canvas; Based on the subject and the parameters, the multiple frames of images are fused to obtain a first image.

4. The method according to claim 1, characterized in that The step of inputting the first image into an image expansion model to obtain a second image output by the image expansion model includes: Acquire audio data corresponding to the dynamic photo; Based on the audio data, determining a second extension element; generating prompt information based on the second extension element; The first image and the prompt information are input into the image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image, the first extended element, and the second extended element.

5. The method according to claim 4, characterized in that The determining, based on the audio data, a second extension element comprises: Based on the audio data, determining at least one candidate element; displaying at least one option corresponding to the at least one candidate element; receiving a third input from a user for a target option among the at least one option; In response to the third input, a second extended element among the at least one candidate element is determined.

6. An image generating device, characterized in that: The device comprises: A receiving unit, configured to receive a first input of a dynamic photo by a user, wherein the dynamic photo includes a plurality of frames of images; a fusion unit, configured to fuse the multiple frames of images in response to the first input to obtain a first image; The image expansion unit is used to input the first image into the image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image and a first extension element, and the first extension element is generated by the image expansion model based on the image content of the first image.

7. The device according to claim 6, characterized in that The fusion unit is further used for: In response to the first input, determining a subject in the dynamic photo; Based on the subject, the multiple frames of images are fused to obtain a first image.

8. The device according to claim 7, characterized in that The fusion unit is further used for: Displaying a setting interface, wherein the setting interface includes a canvas, and the canvas includes the subject; receiving a second input from a user on the setting interface; In response to the second input, obtaining at least one of the following parameters: a position of the subject in the canvas, a size of the subject, and a size of the canvas; Based on the subject and the parameters, the multiple frames of images are fused to obtain a first image.

9. The device according to claim 6, characterized in that The image expansion unit is further used for: Acquire audio data corresponding to the dynamic photo; Based on the audio data, determining a second extension element; generating prompt information based on the second extension element; The first image and the prompt information are input into the image expansion model to obtain a second image output by the image expansion model, wherein the second image includes the image content of the first image, the first extended element, and the second extended element.

10. The device according to claim 9, characterized in that The image expansion unit is further used for: Based on the audio data, determining at least one candidate element; displaying at least one option corresponding to the at least one candidate element; receiving a third input from a user for a target option among the at least one option; In response to the third input, a second extended element among the at least one candidate element is determined.

Images