Interactive image editing method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an interactive image editing method, electronic equipment and a computer readable storage medium, in the method, the electronic equipment displays a to-be-edited image comprising at least one object; in response to the first operation, a target object is selected from the at least one object, the target object in the to-be-edited image is located at a first position, and the shadow of the target object is located at a second position; and displaying the first image in response to the second operation. The first image is an image obtained after the target object and the shadow of the target object in the to-be-edited image are moved according to the second operation. The target object in the first image is located at a third position, the shadow of the target object is located at a fourth position, and the third position is the position indicated by the second operation; when the target object moves, the shadow of the target object moves along with the target object. Thus, when the target object moves, the shadow of the target object moves along with the target object, shadow following is achieved, and the image editing effect is improved.

Description

Technical Field

[0001] This application relates to the field of imaging technology, and more particularly to an interactive image editing method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the increasing popularity and powerful functions of mobile phones and other electronic devices, more and more users are using mobile phones and other electronic devices for image editing.

[0003] Mobile phones and other electronic devices can provide users with user interfaces and tools based on interactive image editing functions, enabling users to easily and conveniently perform image editing operations such as object translation, cropping, 2D rotation, scaling, adjusting color balance, and adding filter effects through interactive means, in order to adjust various aspects of the image such as object position, size, shape, color, brightness, and contrast.

[0004] However, currently, when users perform image editing operations such as translation and rotation on objects in two-dimensional images, the image editing effect on mobile phones and other electronic devices is poor, resulting in a bad user experience. Summary of the Invention

[0005] This application provides an interactive image editing method, an electronic device, and a computer-readable storage medium, which can improve the image editing effect.

[0006] In a first aspect, embodiments of this application provide an interactive image editing method. In this interactive image editing method, an electronic device displays an image to be edited, the image including at least one object; in response to a first operation on the at least one object, a target object is selected from the at least one object, the target object in the image to be edited is positioned at a first position, and the shadow of the target object is positioned at a second position; in response to a second operation on the target object, a first image is displayed. The first image is an image obtained after moving the target object and the shadow of the target object in the image to be edited according to the second operation. The target object in the first image is positioned at a third position, and the shadow of the target object is positioned at a fourth position, the third position being the position indicated by the second operation. The second operation is used to move the target object; when the target object moves, the shadow of the target object moves with the target object.

[0007] As can be seen from the above technical solution, after selecting a target object from at least one object, in response to the second operation for moving the target object, not only is the target object moved from the first position to the third position, but the shadow of the target object also moves along with the target object, so that the shadow of the target object moves from the second position to the fourth position. In this way, by making the shadow of the target object move along with the target object, the image editing effect is improved.

[0008] In one possible implementation of the first aspect, during the process of displaying the first image in response to a second operation on the target object, the electronic device may first extract the target object and its shadow from the image to be edited; then superimpose the target object onto a third position of the second image, and make the shadow of the target object follow the target object to a fourth position of the second image to obtain the first image; and then display the first image. The second image is obtained by removing the target object and its shadow from the image to be edited and filling in a first missing area, where the first missing area is the missing area that appears after removing the target object and its shadow. In this way, by extracting the target object and its shadow from the image to be edited and superimposing them onto the corresponding positions of the second image, the shadow of the target object moves with the movement of the target object, improving the image editing effect.

[0009] In one possible implementation of the first aspect, during the process of extracting the target object and its shadow from the image to be edited, the electronic device can input a first mask of the image to be edited and the target object into a variational autoencoder to obtain the image tensor and mask tensor output by the variational autoencoder. Then, the image tensor and mask tensor are input into a shadow encoder to obtain a second mask output by the shadow encoder. The second mask is a mask of the target object and its shadow. Finally, based on the second mask, the target object and its shadow are segmented from the image to be edited. In this way, after generating a shadow-bearing mask (i.e., the second mask) through the variational autoencoder and the shadow encoder, the target object and its shadow are accurately extracted from the image to be edited using the shadow-bearing mask.

[0010] In one possible implementation of the first aspect, during the process of superimposing the target object onto the third position of the second image and having the shadow of the target object follow the superimposed target object to the fourth position of the second image to obtain the first image, the electronic device can also determine the perspective scaling relationship and foreground / background occlusion relationship of the target object at the third position based on the depth information of the image to be edited. Based on the perspective scaling relationship, the target object and its shadow are scaled down or up. Then, based on the foreground / background occlusion relationship, the target object is superimposed onto the third position of the second image, and its shadow follows the superimposed target object to the fourth position of the second image to obtain the first image. Here, the foreground / background occlusion relationship is used to characterize the occlusion relationship between the target object, its shadow, and other objects.

[0011] When a user moves a target object, the process goes beyond simply moving it to a specified position. It further considers changes in depth of field and occlusion relationships before and after the movement. Based on these changes in depth of field, the target object and its shadow are correspondingly scaled down or up. Then, according to the foreground and background occlusion relationships, the scaled-down or scaled-up target object and its shadow are superimposed onto the image to achieve a better edited image. This further improves the image editing effect.

[0012] In one possible implementation of the first aspect, before superimposing the target object at the third position of the second image and making the shadow of the target object follow the target object superimposed to the fourth position of the second image, the electronic device may also adjust the hue of the target object and the shadow of the target object according to the hue of the third position, since the hue of the target object and the shadow of the target object are different before and after the movement; and fill in the missing elements of the target object.

[0013] When the user moves the target object, the tones of the target object and its shadow are adjusted based on the tones of the area before and after the movement, ensuring consistency between the tones of the target object and its shadow and the tones of the area after movement, resulting in a better image editing effect. Furthermore, when elements are missing from the target object (e.g., due to the target object being occluded by other objects, causing some elements to be missing after extraction), the system proactively fills in the missing elements, further improving the image editing effect. This further enhances the image editing results.

[0014] In one possible implementation of the first aspect, the electronic device can also highlight the associated plane of the target object. During the movement of the target object, if the target object's movement exceeds the associated plane, a new associated plane is generated based on the target object's current position and highlighted. If the target object's movement does not exceed the associated plane, the target object and its shadow can be moved to a specified position. In this way, by highlighting the associated plane of the target object, users can easily identify the plane associated with the target object at its current position, improving the user's image editing experience.

[0015] In one possible implementation of the first aspect, the image to be edited is a two-dimensional image with a single viewpoint, and the target object is in a first pose within the image to be edited. The electronic device can also display a third image in response to a third operation on the target object. The third image is obtained by performing a three-dimensional rotation on the target object in the image to be edited according to the third operation. In the third image, the target object is in a second pose, and the third operation is used to perform a three-dimensional rotation on the target object from the first pose to the second pose.

[0016] By responding to a third operation that performs a three-dimensional rotation on the target object and indicates the desired pose of the target object, the target object in the two-dimensional image is rotated in three dimensions to rotate the target object from the first pose to the desired pose (i.e., the second pose), and the three-dimensional rotated target object is displayed, thereby realizing the image editing effect of three-dimensional rotation of the object in the two-dimensional image and meeting the user's three-dimensional rotation editing needs.

[0017] In one possible implementation of the first aspect, during the process of displaying a third image in response to a third operation on the target object, the electronic device can segment the target object in a first pose from the image to be edited to obtain a fourth image, which is the image obtained after removing the target object from the image to be edited; fill in the second missing region in the fourth image to obtain a fifth image, where the second missing region is the missing region that appears after removing the target object; generate a target object in a second pose based on the target object in the first pose, the pose indicated by the third operation, and the rotation task; superimpose the target object in the second pose and the fifth image to obtain a third image; and display the third image. Thus, by generating a target object in a second pose based on the first pose, the desired pose, and the target object, and superimposing the target object in the second pose onto the image, the effect of three-dimensional rotation of an object in a two-dimensional image is achieved.

[0018] In one possible implementation of the first aspect, during the process of generating a target object in a second pose based on the target object in the first pose, the pose of the third operation instruction, and the rotation task, the electronic device can generate an input image based on the target object in the first pose and a third mask of the target object; generate an embedding vector based on the first pose, the pose of the third operation instruction, and the rotation task; input the input image and the embedding vector into a generation model to obtain the target object in the second pose output by the generation model; the embedding vector is the key and value input of the generation model, and the input image is the query input of the generation model. In this way, generating a target object in the second pose through a generation model facilitates the subsequent implementation of 3D rotation editing effects on objects in a 2D image.

[0019] In one possible implementation of the first aspect, the electronic device can further remove the shadow of the target object in the first pose from the fourth image, that is, remove the shadow of the target object in the original pose from the image. At this time, during the process of superimposing the target object in the second pose and the fifth image to obtain the third image, the electronic device can generate the shadow of the target object in the second pose; the target object in the second pose and its shadow are superimposed on the fifth image to obtain the third image. Thus, when performing a three-dimensional rotation on the target object in the two-dimensional image, removing the shadow of the target object in the original pose from the image, regenerating the shadow of the target object in the desired pose, and superimposing the shadow of the target object in the desired pose and the target object in the desired pose onto the image, ensures that the shadow of the target object changes accordingly during the three-dimensional rotation of the target object, further improving the image editing effect.

[0020] In one possible implementation of the first aspect, the image editing interface displays a rotation angle adjustment control, which is used to adjust the three-dimensional rotation angle of the target object; the third operation is an input operation to the rotation angle adjustment control. Thus, by providing a control for three-dimensional rotation in the editing interface, users can easily rotate the target object in three dimensions, improving the user's image editing experience.

[0021] In a second aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the first aspects above.

[0022] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0023] Fourthly, embodiments of this application provide a chip system including a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the method described in any of the first aspects above. The chip system may be a single chip or a chip module composed of multiple chips.

[0024] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in any of the first aspects above.

[0025] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0026] Figure 1AA schematic diagram of image editing provided for an embodiment of this application;

[0027] Figure 1B Another image editing schematic diagram provided for an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of the structure of the electronic device 200 provided in the embodiments of this application;

[0029] Figure 3A This is a schematic diagram illustrating the selection of an image to be edited, provided in an embodiment of this application.

[0030] Figure 3B This is a schematic diagram illustrating the selection of a target object via touch operation, provided as an embodiment of this application.

[0031] Figure 3C A schematic diagram illustrating the entry into translation mode provided in an embodiment of this application;

[0032] Figure 3D Another schematic diagram illustrating the entry into translation mode provided in this application embodiment;

[0033] Figure 3E A schematic diagram of a movable object provided in an embodiment of this application;

[0034] Figure 4A This is a schematic diagram of an image processing process provided in an embodiment of this application;

[0035] Figure 4B This is a schematic diagram of another image processing procedure provided in an embodiment of this application;

[0036] Figure 4C A schematic diagram of the mask generation process with shadows provided in an embodiment of this application;

[0037] Figure 5A This is a schematic diagram of three-dimensional object movement provided in an embodiment of this application;

[0038] Figure 5B This is a schematic diagram of another image processing procedure provided in an embodiment of this application;

[0039] Figure 6A This is a schematic diagram of the object movement process provided in an embodiment of this application;

[0040] Figure 6B A schematic diagram of an object movement provided in an embodiment of this application;

[0041] Figure 6C This is a schematic diagram of an object before it is moved, provided in an embodiment of this application.

[0042] Figure 6DThis is a schematic diagram of an image of an object after it has been moved, provided in an embodiment of this application.

[0043] Figure 7 This is a schematic diagram of a two-dimensional rotation image editing operation provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram comparing two-dimensional and three-dimensional rotations provided in the embodiments of this application;

[0045] Figure 9A A schematic diagram of entering rotation mode provided in an embodiment of this application;

[0046] Figure 9B Another schematic diagram illustrating the entry into rotation mode provided in this application embodiment;

[0047] Figure 9C This is a schematic diagram of the rotation angle input provided in an embodiment of this application;

[0048] Figure 10A This is a schematic diagram of the image processing process provided in the embodiments of this application;

[0049] Figure 10B This is a schematic diagram of the rotation angle input provided in an embodiment of this application;

[0050] Figure 10C A schematic diagram illustrating the process of generating a target object from a rotated perspective, as provided in an embodiment of this application.

[0051] Figure 10D This is a schematic diagram of the LDM model calculation process provided in the embodiments of this application;

[0052] Figure 10E A schematic diagram of the model calculation process provided in the embodiments of this application;

[0053] Figure 11 This is a schematic diagram illustrating shadow generation during the 3D editing process provided in an embodiment of this application;

[0054] Figure 12 This is a schematic diagram of the image editing operation process provided in the embodiments of this application;

[0055] Figure 13 This is a flowchart illustrating an interactive image editing method provided in an embodiment of this application. Detailed Implementation

[0056] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application.

[0057] When mobile phones and other electronic devices enter image editing mode, they can respond to the user's dragging operation on objects in the image, moving the objects to a specified position to achieve the image editing effect of moving objects in the image.

[0058] For example, see Figure 1A The illustrated embodiment of this application provides an image editing schematic diagram. The image viewing interface 101 of a mobile phone 100 includes a share control, an edit control 103, a delete control, and an image to be edited. The image viewing interface 101 can be the interface of image editing software, which can be, but is not limited to, a gallery application (or photo album application). The image to be edited includes an object 102. The user can click the edit control 103 to instruct the mobile phone 100 to enter image editing mode. The mobile phone 100 responds to the click operation on the edit control 103, enters image editing mode, and displays the image editing interface 104. The image editing interface 104 includes controls for drawing, mosaic, watermarking, stickers, trimming, filters, and adjustments.

[0059] In image editing mode, after receiving a long-press operation on object 102, the phone 100 responds by displaying object 102 as selected. That is, the user selects object 102 as the target object through a long-press operation. The target object refers to the image to be edited, and in this case, it refers to the object that needs to be moved. Figure 1A In the example, object 102 is selected by using a dashed line to indicate that object 102 is selected.

[0060] After selecting object 102, the user can drag object 102 to a specified position using their finger to move object 102 from its initial position to the specified position. For example... Figure 1A As shown, on the image editing interface 104, the user presses and holds the selected object 102 with their finger, drags it to the right to the specified position, and then releases the finger to pan the object 102 to the right. The phone 100 responds to the user's dragging operation and displays the panned object 102 on the image editing interface 104. After panning the object 102 to the right, the area where the object 102 was originally located will show a missing portion of the image, such as... Figure 1A The missing region 105 is shown in the figure. The outline of the missing region 105 is roughly the same as the outer outline of the object 102.

[0061] In related technologies, when a user moves a target object in an image, the electronic device only moves the target object to the designated position without processing its shadow, leaving the shadow in its original position. This results in an unnatural and unrealistic relative position between the target object and its shadow (e.g., the distance between them is too large), leading to poor image editing results.

[0062] When an image is captured, shadows of varying shapes are cast at corresponding locations based on the relative positions of the light source and the subject. For example, when the sun shines from the left side of the subject, a shadow will be cast on the right side of the subject.

[0063] For example, see Figure 1B The illustrated embodiment of this application provides another image editing schematic diagram. After entering image editing mode, the mobile phone 100 displays an image editing interface 106. The image editing interface 106 includes an object 107 and a shadow 108. The shadow 108 is the shadow of the object 107.

[0064] The user long-presses object 107 with their finger on the image editing interface 106 to select object 107 as the target object (i.e., the object to be moved). The mobile phone 100 responds to the long-press operation on object 107 by displaying object 107 as selected. Figure 1B In the example, object 107 is selected with a dashed line to indicate that object 107 is selected.

[0065] After selecting object 107, the user presses and holds object 107 with their finger, drags it to the desired location, and then releases the finger. Phone 100 responds to the user's dragging action by displaying the moved object 107, as shown below. Figure 1B The image editing interface is shown in Figure 109.

[0066] The initial position is taken from the position of object 107 in image editing interface 106. In image editing interface 109, the position of object 107 is different from the initial position, and a missing area 110 appears at the initial position. The position of shadow 108 is the same in image editing interfaces 106 and 109. That is, when object 107 moves, shadow 108 does not move with object 107, but remains in its original position.

[0067] As can be seen from the image editing interface 109, the distance between object 107 and shadow 108 is too large, making the edited image unrealistic and unnatural.

[0068] To address the aforementioned issues, this application provides an interactive image editing solution. When a user moves a target object in an image, the electronic device not only moves the target object but also makes its shadow move along with it, achieving shadow following. This makes the relative position between the target object and its shadow more natural and realistic after image editing, improving the image editing effect.

[0069] The interactive image editing solution provided in this application can be applied to electronic devices such as mobile phones, tablets, and in-vehicle systems. No specific type or structure of the electronic device is limited herein.

[0070] For example, Figure 2 A schematic diagram of the structure of electronic device 200 is shown. Electronic device 200 may include, but is not limited to, processor 210, memory 220, and display screen 230. Optionally, electronic device 200 may also include touch sensor 240, audio module 250, and microphone 260. Touch sensor 240 may be disposed on display screen 230, and touch sensor 240 and display screen 230 together form a touch screen, also known as a "touchscreen".

[0071] The structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device 200 is specifically a mobile phone, it may also include a universal serial bus (USB) interface, a charging management module, a power management module, a camera, a battery, an antenna, a mobile communication module, a wireless communication module, a speaker, a receiver, a headphone jack, sensors, buttons, a motor, an indicator, and a subscriber identification module (SIM card) interface, etc.

[0072] Processor 210 may include one or more processing units, such as an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), and a controller. These different processing units may be independent devices or integrated into one or more processors.

[0073] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0074] The processor 210 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), and a general-purpose input / output (GPIO) interface, etc.

[0075] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C buses. The processor 210 can couple to the touch sensor 240 and the camera, etc., through different I2C bus interfaces. For example, the processor 210 can couple to the touch sensor 240 through the I2C interface, enabling the processor 210 and the touch sensor 240 to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 200.

[0076] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display screen 230 and cameras. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 210 and the camera communicate via the CSI interface to enable the electronic device 200 to capture images. The processor 210 and the display screen 230 communicate via the DSI interface to enable the electronic device 200 to display images.

[0077] The GPIO interface can be configured via software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 to a camera and display 230, etc.

[0078] The interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0079] Electronic device 200 implements display functions through a GPU, display screen 230, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 230 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0080] The display screen 230 is used to display images, videos, etc. The display screen 230 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or N display screens 230, where N is a positive integer greater than 1.

[0081] Memory 220 can be used to store computer executable program code, including instructions. Memory 220 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as image playback function), etc. The data storage area may store data created during the use of electronic device 200. In addition, memory 220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in memory 220 and / or instructions stored in memory disposed in the processor.

[0082] Touch sensor 240, also known as a "touch device," is used to detect touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 230. For example, when a user is editing an image on their phone, dragging an object in the image on the phone's touchscreen, the phone displays visual output related to the dragging operation on display screen 230.

[0083] Electronic device 200 can implement audio functions, such as recording, through audio module 250, microphone 260, and application processor.

[0084] The audio module 250 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 250 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 250 may be located in the processor 210, or some functional modules of the audio module 250 may be located in the processor 210.

[0085] Microphone 260, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Electronic device 200 may be equipped with at least one microphone 260. During image editing in this embodiment, electronic device 200 can acquire user speech through at least one microphone 260 and identify the object specified by the user's speech as the target object.

[0086] The following uses electronic device 200 as an example to illustrate the interactive image editing scheme provided in the embodiments of this application.

[0087] The interactive image editing scheme provided in this application embodiment may include: selecting an image to be edited; selecting a target object; inputting a movement position; moving the target object to generate a new image, etc. Each part will be described in detail below.

[0088] Step 1: Select the image to be edited:

[0089] The electronic device 200 can display one or more two-dimensional images on the display screen 230, allowing the user to select the image to be edited. After the user inputs an operation to select the image to be edited into the electronic device 200, the electronic device 200 responds to the user's input and determines the image to be edited. The image to be edited refers to the image that needs to be edited, and this image is usually a two-dimensional image with a single viewpoint.

[0090] For example, see Figure 3A The illustration shown is a schematic diagram of image selection to be edited provided in this embodiment of the application. The main interface 301 of the mobile phone 300 displays icons for various applications such as the gallery 302 and the camera 303. Users can select one image from multiple images in the gallery 302 as the image to be edited, or they can select the image to be edited from the camera 303.

[0091] When a user selects an image to be edited from the gallery 302, the mobile phone 300 responds to the user's click on the gallery icon 302 and displays the album interface 304. The album interface 304 includes multiple cards such as a card 305 for all photos and a video card. In response to the user's click on card 305, the mobile phone 300 displays the all-photos interface 306. The all-photos interface 306 includes multiple image cards such as image 307 and image 308. After the user selects image 307 as the image to be edited by clicking on the image 307 card, the mobile phone 300 responds to the click on image 307 and displays the image preview interface 309. The image preview interface 309 displays image 307.

[0092] When a user selects an image to be edited from the camera 303, the mobile phone 300 responds to the user's tap on the camera 303 icon and displays a photo preview interface 310. The photo preview interface 310 includes a viewfinder and a thumbnail control 311. At this time, the mobile phone 300 can respond to the user's tap on the shutter button, take a picture to obtain an image 307, and display a thumbnail of the image 307 on the thumbnail control 311.

[0093] In response to a user's click on the thumbnail control 311, the mobile phone 300 displays an image preview interface 309. The thumbnail control 311 is a thumbnail of image 307, therefore image 307 is displayed in the image preview interface 309. At this time, the user selects image 307 as the image to be edited by clicking on the thumbnail control 311.

[0094] Besides being able to, as Figure 4A The image to be edited is determined in the manner shown. Users can also instruct the mobile phone 100 on the image to be edited through voice interaction, text input, or gesture input.

[0095] It should be noted that besides being able to enter image editing mode via the gallery and camera apps, electronic device 200 can also be triggered to enter image editing mode in other ways. For example, when setting a wallpaper, if the user needs to edit the wallpaper, they can also trigger electronic device 200 to enter image editing mode to edit the wallpaper.

[0096] Step 2: Select the target object:

[0097] The image to be edited may include one or more objects. Objects can be, but are not limited to, people, trees, and items. For example, the object in image 307 is a cup.

[0098] In some embodiments, a user can identify a target object from multiple objects in an image to be edited via touch operations. This target object is the object the user needs to edit (rotate or translate). There can be one or more target objects. Touch operations can include, for example, single-click, double-click, or long-press.

[0099] For example, see Figure 3B The schematic diagram shown is based on an embodiment of this application that provides a method for selecting a target object via touch operation. Figure 3A In the scene shown, the image preview interface 309 includes a display area for an image 307 and an editing control 314. The image 307 includes an object 312 and a shadow 313. The shadow 313 is the shadow of the object 312.

[0100] In response to a user's click on the editing control 314, the mobile phone 300 enters image editing mode and displays the image editing interface 315. The image editing interface 315 includes controls for drawing, trimming, and mosaicking, as well as a display area for the image 307.

[0101] To select object 312 in image 307 as the target object, the user taps, double-taps, or long-presses the screen area where object 312 is located. The phone 300 responds to the user's action by displaying object 312 as selected. Figure 4B In the image, a dotted line is used to select object 312 to indicate that object 312 is selected. Of course, the mobile phone 300 can also use other display methods (such as flashing or highlighting object 312) to indicate that object 312 is selected. Alternatively, after determining the target object specified by the user, the mobile phone 300 may choose not to display the target object as selected.

[0102] In other embodiments, the electronic device 200 may display a text input box on the display screen 230 for the user to input text. The electronic device 200 determines the target object selected by the user based on the text entered.

[0103] For example, in Figure 3B In the scenario shown, after displaying the image editing interface 315, the mobile phone 300 can display a text input box in the image editing interface 315; the user can enter "cup" in the text input box to select object 312 as the target object. At this time, the mobile phone 300 determines that the user has selected object 312 as the target object based on the text "cup" entered by the user, and then displays object 312 as selected in the image editing interface 315.

[0104] In some other embodiments, the electronic device 200 can interact with the user via voice through the audio module 250 and microphone 260, and collect the user's voice input to determine the target object selected by the user based on the user's voice.

[0105] For example, in Figure 3B In the scenario shown, after displaying the image editing interface 315 on the mobile phone 300, the user can input the voice command "select cup" to select object 312 as the target object. After the mobile phone 300 collects the voice command "select cup", it determines that the user has selected object 312 as the target object based on the voice data, and then displays object 312 as selected on the image editing interface 315.

[0106] After selecting the target object, the electronic device 200 performs semantic segmentation on the image to segment out the target object. Alternatively, the electronic device 200 can first perform image semantic segmentation on the image to be edited, segmenting out all objects in the image, and then the user selects the target object. For example, in... Figure 3B In the image editing mode, after entering the image editing mode, the mobile phone 300 automatically performs image semantic segmentation on the image 307, segments out the object 312, and prompts the user to select the object on the image editing interface 315.

[0107] Optionally, after selecting a target object, the electronic device 200 can not only display the target object as selected, but also display the target object's shadow as selected. In this case, after identifying the target object, the electronic device 200 can automatically recognize the target object's shadow and select both the target object and its shadow. This allows the user to see the target object and its shadow that are about to be moved, resulting in a better user experience.

[0108] Step 3: Enter the movement location:

[0109] After selecting a target object, the user can input their movement position into the electronic device 200 through human-computer interaction. The movement position input by the user typically represents the desired location of the target object. The electronic device 200, in response to the user's movement position input, determines the movement position of the target object.

[0110] In some embodiments, the electronic device 200 may provide a panning mode to the user, allowing the user to move a target object to a specified location by inputting the movement position.

[0111] For example, see Figure 3C The diagram shown is a schematic representation of entering the translation mode provided in an embodiment of this application, based on... Figure 3BIn the scenario shown, after selecting object 312 on the image editing interface 315, the user can long-press, tap, or double-tap the area containing object 312. In response to these actions, the mobile phone 300 displays a move option 316 and a rotate option 317. Move option 316 is used to enter move mode, and rotate option 317 is used to enter rotate mode. Once in move mode, the user can translate objects in the 2D image; once in rotate mode, the user can rotate objects in the 2D image in three dimensions.

[0112] Of course, the electronic device 200 can also display the move option 316 and the rotate option 317 when the user long presses, clicks or double-clicks the object 315 to select the object 312 as the target object.

[0113] After the user clicks the move option 316, the mobile phone 300 responds to the click operation on the move option 316, enters the pan mode, and displays the move option 316 and the rotate option 317 on the image editing interface 315.

[0114] For example, see also Figure 3D The diagram shown is another schematic diagram of entering the translation mode provided by the embodiment of this application, based on... Figure 3B In the scenario shown, after selecting object 312 on the image editing interface 315, the user can click on the more controls 318 on the image editing interface 315. The mobile phone 300 responds to the click operation on the more controls 318 and displays interface 319. Interface 319 includes a move option 316 and a rotate option 317. The user can enter rotation mode by clicking the rotate option 317; and enter move mode by clicking the move option 316.

[0115] Of course, in some other embodiments, the electronic device 200 may not provide a panning mode. In this case, after the electronic device 200 enters the image editing mode, the user can move the target object.

[0116] When inputting a movement position into the electronic device 200, the user can input the movement position through touch operations such as dragging. For example, the user presses and holds the target object with their finger, drags it a certain distance, and then releases it to place the target object in a specified position; the electronic device 200 obtains the movement position of the target object based on the user's dragging operation.

[0117] For example, see Figure 3E The diagram shown is a schematic representation of a movable object provided in an embodiment of this application, based on... Figure 3B , Figure 3C or Figure 3DIn the scene shown, in the image editing interface 315, object 312 is located at position A, and shadow 313 is located at position B. The user presses and holds object 312 with their finger, slides it in the first direction, and releases object 312 after sliding it to position C, thus placing object 312 at position C, as specifically shown in the image editing interface 320. The first direction is as follows: Figure 3E As shown.

[0118] Since the shadow 313 moves along with the object 312, after the user places the object 312 at position C, the shadow 313 also moves to position D along with the object 312.

[0119] In response to the drag operation on object 312, mobile phone 300 determines that the user-inputted movement position is position C.

[0120] Of course, users can also input their location to the electronic device 200 through gesture input, text input, or voice input, and the electronic device 200 will respond to the user's gesture input, text input, or voice input to obtain the location of the target object.

[0121] Step 4: Move the target object to generate a new image:

[0122] Electronic device 200 segments the target object and its shadow from the image to be edited; the segmented target object is then overlaid onto the moving position, and the shadow of the target object follows the overlay to the corresponding position, resulting in a new image. In the new image, the target object is positioned at the user-specified moving position, and the shadow of the target object also moves to the corresponding position along with the target object.

[0123] For example, see Figure 4A The illustrated diagram shows an image processing procedure provided in an embodiment of this application, based on... Figures 3A-3E In this scenario, the image selected by the user to be edited is image 307, and the target object selected is object 312.

[0124] Mobile phone 300 performs image semantic segmentation on image 307 to obtain object 312, shadow 313, and image 321. Image 321 is the image obtained after removing object 312 and shadow 313 from image 307. Image 321 includes a missing region 322, which is the original location of object 312 and shadow 313. That is, after removing object 312 and shadow 313 from image 307, there will be missing images in the areas where object 312 and shadow 313 originally were.

[0125] After obtaining image 321, mobile phone 300 completes the missing region 322 in image 321 to obtain image 323; then, object 312 and shadow 313 are superimposed on the corresponding positions in image 323 to obtain image 324; image 324 is then displayed, as shown. Figure 3E The image editing interface 320 is shown in the image editing interface.

[0126] In this case, object 312 and shadow 313 can be superimposed onto image 323 as a whole. Based on the user's drag operation, mobile phone 200 can determine the moving position as position C; when object 312 is superimposed onto position C in image 323, since object 312 and shadow 313 are as a whole, shadow 313 will also be superimposed onto position D in image 323 along with object 312.

[0127] For example, see also Figure 4B The illustrated diagram shows another image processing procedure provided by an embodiment of this application, based on... Figures 3A-3E In this scenario, the image selected by the user to be edited is image 307, and the target object selected is object 312.

[0128] Mobile phone 300 performs semantic segmentation on image 307 to obtain object 312, shadow 313, and image 321; object 312 and shadow 313 are superimposed onto their corresponding positions in image 321 to obtain image 325. Since missing region completion was not performed, missing region 322 still exists in image 325; missing region 322 in image 325 is completed to obtain image 324; image 324 is then displayed, as shown... Figure 3E The image editing interface 320 is shown in the image editing interface.

[0129] By comparison Figure 4A and Figure 4B It can be seen that, Figure 4A First, fill in the missing areas, then overlay the layers; Figure 4B First, overlay the image, then fill in the missing areas. In contrast, filling in the missing areas first and then overlaying the image ensures that the user sees the completed image when moving object 312, resulting in a better user experience.

[0130] It should be noted that the execution order of steps 1, 2, 3 and 4 in the above embodiments can be set according to actual needs, and the order of execution is not limited here. For example, the above description uses the example of executing step 1 first and then step 2, but in other embodiments, step 1 can be executed first and then step 2, or steps 1 and 2 can be executed simultaneously.

[0131] In some alternative embodiments, the electronic device 200 may first generate a shaded mask, and then extract the target object and its shadow from the image to be edited based on the shaded mask. The shaded mask includes a mask of the target object and a mask of the target object's shadow.

[0132] For example, see Figure 4C The schematic diagram of the mask generation process with shadow provided in the embodiment of this application shows that the image 307 and the mask 361 are input to the variational auto-encoder (VAE) to obtain the image tensor and mask tensor output by the VAE; the image tensor and mask tensor are input to the shadow encoder to obtain the mask 362 output by the shadow encoder.

[0133] Mask 361 is the mask image of object 312. Mask 362 is a mask with shadow, which can be a mask for both object 312 and shadow 313.

[0134] After obtaining the mask 362, the mobile phone 300 can extract objects 312 and 313 from the image 307 based on the mask 362.

[0135] A mask with shadows can accurately represent the outline of the target object and its shadow, thus enabling the accurate extraction of the target object and its shadow from the image to be edited.

[0136] In some alternative embodiments, the electronic device 200 can use a latent diffusion model (LDM) to automatically fill in missing regions. For example, the electronic device 200 inputs an image including the missing regions and a mask of the missing regions into the LDM model; that is, the missing regions are input as the mask, and the image including the missing regions is input as the image. The LDM model automatically fills in the missing regions, completing them, and outputs the completed image.

[0137] exist Figure 4A In the process, the mobile phone 300 can input the mask of image 321 and missing region 322 into the LDM model to obtain image 323 output by the LDM model.

[0138] exist Figure 4B In the process, the mobile phone 300 can input the mask of image 325 and missing region 322 into the LDM model to obtain image 324 output by the LDM model.

[0139] Electronic device 200 can also input a shaded mask and the original image into the LDM model to obtain a completed image output by the LDM model. In this case, the completed image does not include missing areas and the target object and its shadow have been removed. For example, mobile phone 300 can input mask 362 and image 307 into the LDM model to obtain image 323 output by the LDM model; then, object 312 and shadow 313 are superimposed onto the corresponding positions of image 323 to obtain a new image.

[0140] It is worth noting that in some embodiments, the electronic device 200 may not fill in the missing areas. However, in contrast, the electronic device 200 automatically fills in the missing areas after the user moves the target object, which can improve the user experience.

[0141] In the above embodiment, the electronic device 200 extracts the target object and its shadow from the image to be edited, and superimposes the target object and its shadow onto a specified position in the image. This allows the shadow of the target object to move along with the target object, making the relative position between the target object and its shadow more natural and realistic during image editing, thus improving the image editing effect.

[0142] In other embodiments, the electronic device 200 can also regenerate the shadow of the target object and overlay the regenerated shadow and the target object onto a designated position in the image. In this way, after the target object moves to the designated position, its shadow will also appear in the corresponding position, resulting in a more natural and realistic relative position between the target object and its shadow, and a better image editing effect.

[0143] At this point, the electronic device 200 can extract the target object from the image to be edited, remove the shadow of the target object, fill in the missing areas in the image, generate the shadow of the target object, and overlay the target object and the generated shadow of the target object onto the image after the missing areas are filled in to obtain a new image.

[0144] In some embodiments, before overlaying the target object and its shadow onto a designated position in the image, the electronic device 200 may also determine whether to reduce or increase the size of the target object and its shadow based on the depth of field before and after the movement; determine whether to adjust the tone of the target object and its shadow based on the tone of the target object before and after the movement; determine whether to change the occlusion relationship between the target object, its shadow, and other objects based on the foreground and background occlusion relationship at the moved position; and fill in missing elements for the target object.

[0145] When the depth of field of the target object is inconsistent between its position before and after movement, the electronic device 200 can reduce or increase the size of the target object and its shadow. The reduction or increase ratio can be determined based on the difference in depth of field before and after movement. The target object and its shadow can be reduced or increased as a whole. This ensures that the size ratio of the target object and its shadow after movement matches the depth of field of its current location, avoiding unrealistic and unnatural appearances due to mismatched size ratios.

[0146] When the color tone of the target object is inconsistent between its position before and after movement, the electronic device 200 can adjust the color tone of the target object and its shadow based on the color tone of the position after movement, so that the color tone of the target object and its shadow matches the color tone of the area after movement. This ensures that the color tone of the target object and its shadow matches the surrounding area after movement, preventing unrealistic and unnatural appearances due to excessive color differences.

[0147] The electronic device 200 can determine the foreground and background occlusion relationship of the target object after it has been moved, based on the depth information of the image to be edited; according to the foreground and background occlusion relationship, it displays the target object, its shadow, and other objects accordingly. This makes the edited image more realistic and natural, resulting in better image editing effects.

[0148] Element padding refers to completing missing parts of a target object or its shadow. In some cases, if a target object or its shadow was partially obscured by other objects before being moved, these obscured areas will be missing after the target object or its shadow is extracted. In this situation, the electronic device 200 can actively pad the missing elements in the target object or its shadow. This allows the target object or its shadow to be superimposed onto the specified location in the image more naturally and realistically, resulting in better image editing effects.

[0149] After scaling down or enlarging the target object and its shadow, adjusting the tone, and filling in the elements, the electronic device 200 can overlay the target object and its shadow onto a specified position in the image based on the occlusion relationship between the foreground and background.

[0150] For example, see Figure 5A The illustrated diagram shows a three-dimensional movement of an object provided in an embodiment of this application, based on... Figure 3AIn this scenario, the user selects image 308 as the image to be edited. After the phone 300 enters image editing mode, it displays image editing interface 326. Image editing interface 326 includes image 308. Image 308 includes objects 327, 328, and 329. Object 328 is a cup with circular spots and a lid, object 327 is a cup with black and white stripes, and object 329 is a cup with checkered spots and a lid.

[0151] To select the black and white striped cup (object 327) in image 308 as the target object, the user can single-click, double-click, or long-press the screen area where object 327 is located. The phone 300 responds to the user's action and selects object 327 as the target object. Alternatively, the user can press and hold object 327 with their finger, slide it to position E, and release their finger to place object 327 at position E. The phone 300, responding to the user's movement of object 327, displays the image editing interface 330. The image processing process performed by the phone 300 based on the user's actions can be found in [reference needed]. Figure 5B .

[0152] See Figure 5B The illustrated diagram shows another image processing procedure provided by an embodiment of this application. The mobile phone 300 segments the image 308 to obtain an object 327 and an image 331. Image 331 includes a missing region 332. The missing region 322 is the missing region that appears after removing the object 327 from the image 308. The missing region is then filled in on the image 331. Based on the depth-of-field difference of the object 327's position before and after movement, the object 327 is magnified. The magnified object 327 is superimposed onto the image after the missing region is filled in to obtain image 333. Image 333 is then displayed on the image editing interface 330.

[0153] In the image editing interface 330, object 327 is located at position E, and object 327 at position E is larger than object 327 in the image editing interface 326.

[0154] In this embodiment, when the target object moves, not only the movement in the xy plane but also the movement in the z-axis direction is considered. The z-axis direction can refer to the depth direction of the image. In this way, the target object can move in the xyz three-dimensional space, achieving three-dimensional object movement. Compared to two-dimensional movement (movement within the xy plane), this embodiment achieves a more realistic and natural editing effect with three-dimensional object movement.

[0155] exist Figure 5BIn this process, the mobile phone 300 magnifies the object 327 and overlays it onto the image based on the depth difference between the object 327's position before and after its movement. In other embodiments, the mobile phone 300 can also perform element completion, tone adjustment, overlay based on foreground and background occlusion relationships, and make the shadow of the object 327 move with the object 327. For example, if a part of the object 327 is occluded by the object 328 in image 326, after the mobile phone 300 extracts the object 327, the part of the object 327 that was originally occluded by the object 328 will be missing. In this case, the mobile phone 300 can automatically fill in the missing part of the object 327. Another example is that the mobile phone 300 determines the foreground and background occlusion relationship at position E based on the depth information at position E. At this point, based on the foreground and background occlusion relationship, it can be seen that part of object 327 will be in front of part of object 329, that is, part of object 327 will occlude part of object 329. At this time, object 329 can be superimposed on the corresponding position in the image first, and then object 327 can be superimposed on the corresponding position in the image, so that part of object 327 and part of object 329 form an occlusion relationship, with object 327 in front and object 329 behind.

[0156] In some embodiments, during object movement, the electronic device 200 can automatically identify the plane associated with the currently moving object and highlight the associated plane, making it easier for users to understand the object's movement and improving the image editing experience. The following describes... Figure 6A and Figure 6B An example will be provided. Figure 6A This is a schematic diagram of the object movement process provided in an embodiment of this application. Figure 6B This is a schematic diagram of an object movement process provided in an embodiment of this application.

[0157] exist Figure 6A In the process, the electronic device 200 selects the image to be edited, then selects a specific object in the image, thus determining the target object; selects the movement mode based on the menu, thus entering the movement mode; and generates an associated planar schematic diagram based on the object (i.e., the target object).

[0158] See Figure 6B It may include, but is not limited to, the following steps:

[0159] Step S601: Input the image to be edited.

[0160] Once the user selects the image to be edited, the electronic device 200 can respond to the user's operation and identify the selected image as the image to be edited, thereby obtaining the image to be edited.

[0161] It is understood that step S601 can be step 1 in the above embodiment. For details, please refer to the relevant content of step 1 in the above embodiment, which will not be repeated here.

[0162] Step S602: Select a specific object in the image.

[0163] After acquiring the image to be edited, the electronic device 200 can, in response to the user's object selection operation, determine the target object (i.e., a specific object) specified by the user. For example, based on... Figure 3A In the scenario shown, the mobile phone 300 responds to the user's operation by selecting image 308 as the image to be edited and determining object 327 in image 308 as the target object.

[0164] It is understood that step S602 can be step 2 in the above embodiment. For details, please refer to the relevant content of step 2 in the above embodiment, which will not be repeated here.

[0165] Step S603: Select the movement mode based on the menu.

[0166] For example, based on Figure 3A In the scenario shown, mobile phone 300 displays an options menu in response to an operation on object 327. This options menu includes a move option 316 and a rotate option 317. Mobile phone 300 enters move mode in response to a tap on move option 316.

[0167] Step S604: Generate a schematic diagram of the associated plane based on the object.

[0168] After entering the mobile mode, the electronic device 200 can determine the plane where the object (i.e., the target object) is located, and designate the plane where the target object is located as the associated plane and highlight it.

[0169] For example, after entering mobile mode, the mobile phone 300 determines that the associated plane of object 327 is plane 336 and highlights plane 336. Figure 6B The plane 336 is highlighted by a diagonal line.

[0170] Step S605: Control the object and move it.

[0171] After the electronic device 200 enters the movement mode, the user can control the object (i.e. the target object) to move, and the electronic device 200 can respond to the user's movement operation and move the object.

[0172] Step S606: Determine whether the movement range exceeds the associated plane. If yes, proceed to step S607; if no, proceed to step S608.

[0173] After highlighting the plane associated with the target object, the electronic device 200 can respond to the user's movement operation on the target object and control the object to move. During the movement of the target object, the electronic device 200 determines in real time whether the current position of the target object is within the associated plane, that is, whether the movement range exceeds the associated plane.

[0174] Step S607: Based on the location of the object, regenerate the associated planar schematic diagram. Then return to step S605.

[0175] If it is determined that the movement range exceeds the associated plane, that is, the target object is not currently located within the associated plane, the electronic device 200 will redetermine the associated plane of the target object based on the current location of the target object, and regenerate the associated plane diagram, that is, continue to highlight the new associated plane.

[0176] Step S608: Based on the target position of the moving object, complete shadow following, front and back occlusion, element completion and tone adjustment, etc.

[0177] If the movement range is determined not to exceed the associated plane, the electronic device 200 performs operations such as shadow following, foreground / background occlusion, element completion, and tone adjustment based on the current position of the target object. Foreground / background occlusion refers to the electronic device 200 overlaying the target object onto the image based on the foreground / background occlusion relationship when generating the image after the movement. Shadow following means that the shadow of the target object moves with the movement of the target object. The specific processes of shadow following, foreground / background occlusion, element completion, and tone adjustment can be found in the above embodiments and will not be repeated here.

[0178] To better illustrate the technical solutions provided in the embodiments of this application, the following is combined with... Figure 6C and Figure 6D An example will be provided. Among them, Figure 6C This is a schematic diagram of the object before it is moved, provided in an embodiment of this application. Figure 6D This is a schematic diagram of an image after the object has been moved, as provided in an embodiment of this application.

[0179] like Figure 6C As shown, image 610 includes a chair 611, a striped cup 612, a table 613, a speckled cup 614, a white checkered cup 615, a black cup 615, and a black checkered cup 616.

[0180] Among them, striped cup 612 is on chair 6111 and is shaded. Spotted cup 614, checkered white cup 615, black cup 615, and checkered black cup 616 are all on table 613. In addition, checkered white cup 615 and black cup 615 also include a plate, and the tabletop also includes a folded tablecloth.

[0181] At this moment, the user wants to move the striped cup 612 from the chair 611 onto the table 613. In response to the user's long-press operation on the striped cup 612, the electronic device 200 displays the striped cup 612 as selected and highlights the plane of the chair 611 (not shown in the figure). That is, the associated plane of the striped cup 612 is the plane of the chair 611 at this time.

[0182] After selecting the striped cup 612, the user drags the striped cup 612 to the corresponding position on the table 613 and releases it. The electronic device 200 responds to the user's dragging operation of the striped cup 612, determining whether the movement range of the striped cup 612 exceeds the plane of the chair 611. Furthermore, it uses the method mentioned above to make the shadow of the striped cup 612 move along with the movement of the striped cup 612.

[0183] When the electronic device 200 determines that the striped cup 612 has entered the desktop of the table 613, it highlights the desktop of the table 613 (not shown in the figure), and determines the front and back occlusion relationship based on the depth relationship between the striped cup 612 and other objects, performs tone adjustment and shadow following, and displays image 618.

[0184] In image 618, striped cup 612 is positioned on the tabletop 613, and it partially obscures the checkered white cup 615 and the plate beneath it. Furthermore, the shadow area of ​​striped cup 612 overlaps with a portion of the spotted cup 614. Electronic device 200 can also adjust the tone of striped cup 612 to match the overall tone of the area around table 613. If any elements are missing, element completion can be performed.

[0185] In related technologies, mobile phones and other electronic devices can only achieve image editing effects such as translation and two-dimensional rotation of objects in two-dimensional images, but cannot achieve image editing effects such as three-dimensional rotation of objects in two-dimensional images, thus failing to meet users' three-dimensional rotation editing needs.

[0186] For example, see Figure 7 The illustrated diagram shows a two-dimensional rotation image editing operation based on an embodiment of this application. Figure 1A In the scenario where the phone 100 is in image editing mode and receives a long press operation on object 102, object 102 will be displayed as selected. After selecting object 102, the user can pan or rotate object 102 in two dimensions.

[0187] like Figure 7As shown, after the user rotates the selected object 102 clockwise, the mobile phone 100 responds to the user's operation and displays the clockwise rotated object 102 in the image editing interface 111. After rotating the object 102 clockwise, the area where the object 102 was originally located will show a missing portion of the image, such as... Figure 7 The missing region 112 is shown in the image.

[0188] pass Figure 7 It can be seen that 2D rotation can only rotate objects in a 2D image within a 2D plane, but cannot rotate objects in a 3D plane.

[0189] To better distinguish between two-dimensional rotation and three-dimensional rotation, the following combines... Figure 8 To provide an introduction and explanation. Figure 8 This is a schematic diagram comparing two-dimensional and three-dimensional rotation provided in the embodiments of this application.

[0190] like Figure 8 As shown, the original image 308 includes object 327. The user performs a two-dimensional rotation of object 327 in the xy-plane to obtain an image including the rotated object 327, as shown in image 337. The user also performs a three-dimensional rotation of object 327 in xyz three-dimensional space to obtain an image including the rotated object 327, as shown in image 338.

[0191] By comparing the original images 308, 337, and 338, it can be seen that the viewpoints of object 327 in the original image 308 and object 327 in image 337 are the same, that is, two-dimensional rotation does not change the viewpoint of object 327 in the image; the viewpoints of object 327 in the original image 308 and the cup in image 338 are different, that is, three-dimensional rotation will change the viewpoint of object 327 in the image.

[0192] In response to the aforementioned issues, the interactive image editing solution provided in this application embodiment can also achieve three-dimensional rotation editing effects, allowing users to conveniently perform three-dimensional rotation editing on objects in two-dimensional images through interactive image editing operations in applications such as image galleries, thus meeting users' three-dimensional rotation editing needs.

[0193] The interactive image editing scheme provided in this application embodiment may include: selecting an image to be edited; selecting a target object; inputting a rotation angle; performing three-dimensional rotation on the target object to generate a new image, etc. Each part will be described in detail below.

[0194] Step a: Select the image to be edited.

[0195] Step b: Select the target object.

[0196] The steps of selecting the image to be edited and selecting the target object are similar to those in the object translation editing process, as detailed in steps 1 and 2 of the above embodiment. Figure 3A and Figure 3B The relevant content will not be repeated here.

[0197] It is understood that step a can be steps 1 and S601 in the above embodiments. Step b can be steps 2 and S602 in the above embodiments. In the above embodiments, the object to be moved is selected from the image to be edited, while here the object to be rotated in three dimensions is selected from the image to be edited. That is, the target object here is the object that needs to be rotated in three dimensions.

[0198] Step c, Input rotation angle:

[0199] After selecting a target object, the user can input a rotation angle into the electronic device 200 through human-computer interaction. The rotation angle input by the user typically represents the desired pose of the target object, that is, the pose of the target object after three-dimensional rotation. In response to the user's rotation angle input, the electronic device 200 determines the desired pose of the target object.

[0200] In some embodiments, the electronic device 200 may provide a rotation mode to the user and provide a rotation angle adjustment control in the rotation mode, allowing the user to input the rotation angle and perform three-dimensional rotation operations on the target object through the rotation angle adjustment control.

[0201] For example, see Figure 9A The diagram shown is a schematic representation of entering a rotation mode provided in an embodiment of this application, based on... Figure 3A In the scenario shown, mobile phone 300 responds to the user's operation and selects image 308 as the image to be edited.

[0202] After selecting object 327 on the image editing interface 339, the user can long press or double-tap the area where object 327 is located. In response to long press or double-tap operations on object 327, the mobile phone 300 displays the move option 316 and the rotate option 317.

[0203] The user taps the rotation option 317 to enter rotation mode. The mobile phone 300 responds to the user's tap on rotation option 317 by displaying the rotation mode interface 340. The rotation mode interface 340 includes rotation angle adjustment controls such as a rotation angle bar 341 and a sliding control 342. The sliding control 342 is on the rotation angle bar 341 and can be slid left and right. The leftmost point of the rotation angle bar 341 is -180°, the rightmost point is 180°, and the middle is 0°. The user can input the rotation angle by dragging the sliding control 342 left or right on the rotation angle bar 341, thereby adjusting the pose of the target object.

[0204] For example, see also Figure 9B The diagram shown is another schematic of entering the rotation mode provided by the embodiment of this application, based on... Figure 9A In the scenario shown, after selecting object 327 on the image editing interface 339, the user can click on the "More Controls" 318 on the image editing interface 339. In response to the click on the "More Controls" 318, the mobile phone 300 displays interface 343. Interface 343 includes a move option 316 and a rotate option 317. The user can enter rotation mode by clicking on the rotate option 317. In response to the user's click on the rotate option 317, the mobile phone 300 displays the rotation mode interface 340.

[0205] After the phone 300 enters rotation mode, the user can manually drag the slider 343 to input the rotation angle into the phone 300 as needed. For example, see... Figure 9C The illustrated diagram shows the rotation angle input provided in this embodiment of the application. After the mobile phone 300 displays the rotation mode interface 340, the object 327 is in its original pose, and the sliding control 342 is at 0° by default. The user manually drags the sliding control 342 to the left to position F to input a rotation angle of -30° to the mobile phone 300; the user manually drags the sliding control 342 to the right to position G to input a rotation angle of 120° to the mobile phone 300.

[0206] In other embodiments, the user can also input the rotation angle to the electronic device 200 via text input or voice input, and the electronic device 200 will respond to the user's text input or voice input to obtain the rotation angle of the target object. For example, the electronic device 200 can provide the user with a text box in which the user can input the rotation angle.

[0207] Step d: Perform a 3D rotation on the target object to generate a new image.

[0208] Electronic device 200 acquires the rotation angle input by the user, performs a three-dimensional rotation on the target object based on the rotation angle, obtains the three-dimensional rotated target object, and generates a new image based on the three-dimensional rotated target object.

[0209] For example, such as Figure 9C As shown, when the user inputs a rotation angle of -30°, the mobile phone 300 rotates the object 327 in the original pose (or original viewpoint) of the rotation mode interface 340 counterclockwise by 30° to generate the object 327 in a new viewpoint (or new pose), and generates a new image based on the object 327 in the new viewpoint, which is then displayed on the interface 344. By comparing the interface 344 and the rotation mode interface 340, it can be seen that the pose (or viewpoint) of the object 327 has changed.

[0210] When the user inputs a rotation angle of 150°, the mobile phone 300 rotates the object 327 in the original pose (or original viewpoint) of the rotation mode interface 340 by 120° clockwise, generating the object 327 in a new viewpoint (or new pose), and generating a new image based on the object 327 in the new viewpoint, which is then displayed on the interface 345. By comparing the interface 345 and the rotation mode interface 340, it can be seen that the pose (or viewpoint) of the object 327 has changed.

[0211] It should be noted that the execution order of steps a, b, c, and d in the above embodiments can be set according to actual needs, and the order of execution is not limited here. For example, the above description uses the example of executing step a first and then step b, but in other embodiments, step a can be executed first and then step b, or steps a and b can be executed simultaneously.

[0212] After determining the target object, the electronic device 200 first segments the target object from the image to obtain the target object and the image with the target object removed; then, according to the rotation angle input by the user, it performs a three-dimensional rotation on the target object to obtain the three-dimensional rotated target object; finally, it superimposes the three-dimensional rotated target object and the image with the target object removed to generate a new image, which includes the three-dimensional rotated target object.

[0213] For example, see Figure 10A The illustrated diagram of the image processing process provided in the embodiments of this application is based on... Figure 3A In this scenario, the user selects image 308 to be edited and object 327 as the target object. Electronic device 200 segments image 308 to obtain object 327 and image 346 from the original viewpoint. Image 346 is the image obtained after removing object 327 from image 308. Image 346 includes a missing region 347, which is the original location of object 327. In other words, after removing object 327 from image 308, the original location of object 327 will be missing.

[0214] In specific applications, electronic device 200 can use image semantic segmentation technology to perform image semantic segmentation on image 407 to segment the object 327 from the original viewpoint. More specifically, electronic device 200 can perform segmentation using an image semantic segmentation model. Of course, electronic device 300 can also segment the object 327 from the original viewpoint from image 308 using other methods.

[0215] After obtaining the object 327 from its original viewpoint, the electronic device 200 can generate the object 327 from a 30° rotated viewpoint based on the user-inputted rotation angle. Figure 10A In this example, we'll use a user-inputted rotation angle of 30°. See also... Figure 10B The schematic diagram of the rotation angle input provided in the embodiment of this application shows that the user manually drags the sliding control 342 to the right to position H to input a rotation angle of 30° to the mobile phone 300.

[0216] After obtaining the object 327 from a 30° rotated perspective, the electronic device 200 overlays the object 327 and the image 348 to generate an image 349, which is then displayed. Image 349 includes the selected object 327 from the 30° rotated perspective. For example, the mobile phone 300 displays image 349 on interface 350. Thus, when the user inputs a rotation angle by dragging the slider 342, the mobile phone 300 can display the object 327 from the 30° rotated perspective on interface 350, allowing the user to see the image editing effect in real time and providing a better user experience.

[0217] Image 348 is the image obtained by automatically completing the missing region 347 of image 346.

[0218] In specific applications, when generating a target object in a three-dimensional rotational view based on the rotation angle, the electronic device 200 can obtain the target object in a three-dimensional rotational view by generating a model; it can also generate the target object in a three-dimensional rotational view in other ways, which are not limited here.

[0219] When generating a target object in a 3D rotated view using a generative mode, the electronic device 200 can input the segmented target object in the original view, the desired pose, and the original pose into the generative model to obtain the target object in a 3D rotated view output by the generative model. The desired pose is the pose of the target object as desired by the user, which can be obtained through the rotation angle input by the user. The original pose is the pose of the target object before the 3D rotation.

[0220] The generative model can be, but is limited to, the latent diffusion model (LDM).

[0221] To better illustrate the process of generating a target object in 3D rotation using a generative model, the LDM model will be used as an example below.

[0222] See Figure 10C The illustrated embodiment of this application provides a schematic diagram of the process for generating a target object from a rotated perspective. The input may include the image to be edited, the desired perspective, and the original pose. In this case, the image to be edited is image 308, which includes object 327 (i.e., the target object). The desired perspective (or desired pose) is the viewpoint of the target object after rotation, as desired by the user. The user can input the desired perspective by inputting a rotation angle. For example, if the user inputs a rotation angle of 30° using the rotation angle bar 341 and the sliding control 342, the user's desired perspective will be the 30° rotated viewpoint.

[0223] Object 327 is segmented from image 308, and then stitched with a blank mask image 352 to obtain image 351. The blank mask image 352 is a mask that covers the possible viewpoints of object 327; its shape is not limited, and it represents the image of the rotated viewpoint to be generated. Optionally, the blank mask image 352 can be obtained by enlarging the outer contour of object 327 by a certain proportion. For example, image 327 can be a 1024x512 dimensional image, and the segmented object 327 is a 512x512 dimensional image.

[0224] The rotation task embedding, view embedding, and original pose embedding are concatenated into a composite embedding of the specified length. The rotation task embedding indicates a 3D rotation of the target object. For example, this composite embedding could be a 77x768-dimensional embedding. In this case, 70x768 of the 77x768 dimensions are allocated to the rotation task embedding; 5x768 dimensions are allocated to the view embedding, representing the desired rotation angle; and 2x768 dimensions are allocated to the original pose embedding, representing the original pitch view of the target object.

[0225] The synthesized and stitched image 351 is input into the LDM model 353 to obtain the output result 354 of the LDM model 353. The output result 354 includes objects 327 from multiple viewpoints. For example, each 3° is considered a viewpoint, so 360° would have 120 viewpoints, and the output result 354 would include objects 327 from 120 viewpoints. Based on the objects 327 from multiple viewpoints in the output result 354, a rotatable model 355 of the objects 327 is generated. The rotatable model 355 can rotate 360° to obtain the objects 327 from any viewpoint. That is, by establishing the rotatable model 355 of the objects 327, objects 327 from multiple viewpoints can be obtained.

[0226] Of course, in other embodiments, object 410 can also be 3D modeled to obtain object 410 from various viewpoints. Alternatively, LDM model 353 can output only object 327 from a specified viewpoint.

[0227] The synthetic embedding is used as the key (K) and value (V) input to the LDM model 353, and the image 351 is used as the query (Q) input to the LDM model 353. The key, value, and query inputs are calculated in each layer of the LDM model 353 to generate the object 327 in the rotated view.

[0228] See Figure 10D The diagram shown illustrates the LDM model calculation process provided in this embodiment of the application. The LDM model 353 includes a U-Net model. The U-Net model includes multiple matrices. Figure 10D Each rectangle in the model represents a matrix, and the number at each rectangle indicates its dimension. The synthesis embedding of the rotation task, desired viewpoint, and original pose provides K and V inputs to each rectangle in the U-Net model, while the 1024x512 dimensional image 351 provides Q inputs. The synthesis embedding includes rotation task parameters, desired viewpoint parameters, and original pose parameters. Each matrix in the U-Net model is calculated based on K, V, and Q, outputting object 356. Object 356 is the object as seen from the viewpoint after object 327 is rotated 30° clockwise.

[0229] Each layer in the U-Net model requires computation based on K, V, and Q. For example, see... Figure 10EThe illustrated model calculation process provided in this embodiment of the application, taking the leftmost matrix of the U-Net model (4x224x448) as an example, involves multiplying the 1024x512 dimension image 426 by Wq to obtain the Query value; the rotation task embedding, the desired viewpoint embedding, and the original pose embedding are multiplied by Wk and Wv respectively to obtain the key and value values; the Query value, key value, and value value are then input into each module for calculation. Each layer in the U-Net model performs corresponding calculations based on the K, V, and Q inputs, enabling the model to calculate the target object under the rotated viewpoint based on the user-input rotation angle and the original target object.

[0230] As can be seen from the above, this embodiment of the application, after the user determines the image to be edited and the target object, segments the target object from the image to be edited; then, using generation modes such as LDM, it generates the target object from the rotated viewpoint based on the user-input rotation angle and the target object in the original pose; finally, it superimposes the target object from the rotated viewpoint and the image with the target object removed to obtain a new image, thus achieving the effect of three-dimensional rotation editing of objects in two-dimensional images, and meeting the user's three-dimensional rotation editing needs.

[0231] During 3D rotation editing, missing areas in an image can be automatically filled in. In practical applications, after extracting the target object from the image to be edited, the missing areas in the image can be filled in first, and then the target object from the rotated viewpoint can be superimposed onto the image; alternatively, the target object from the rotated viewpoint can be superimposed onto the image first, and then the missing areas in the image can be filled in.

[0232] In this embodiment, the missing regions in the image can be used as the mask input of the generative model. The image after the target object is extracted, or the image with the rotated viewpoint superimposed, can be used as the image input of the generative model. The generative model is used to automatically fill the missing regions in the new image to obtain the image output by the generative model, which is the final edited image.

[0233] In some embodiments, during the 3D rotation editing process, the electronic device 200 also regenerates the shadow of the target object from the rotated viewpoint and overlays the target object from the rotated viewpoint and its shadow onto the image, making the edited image more natural and realistic.

[0234] For example, see Figure 11The illustration shows a schematic diagram of shadow generation during the 3D editing process provided in this application embodiment. Image 1101 includes object 1102 and shadow 1103. Shadow 1103 is the shadow of object 1102. The user rotates object 1103 in image 1101 in 3D, so the rotation angle is input to electronic device 200. Electronic device 200 extracts object 1102 and shadow 1103 from image 1102, and then fills in the missing areas of the image after extracting the object and shadow to obtain the completed image; based on the extracted object 1102 and the rotation angle, object 1102 is generated from the rotated viewpoint; shadow 1105 is generated based on the incident angle of light in image 1101 and the position of the object; shadow 1105 and object 1102 from the rotated viewpoint are superimposed on the completed image to obtain image 1104.

[0235] By comparing images 1104 and 1101, it can be seen that the object 1102 is viewed from different angles in the two images, and the shape and position of the shadow are also different under different angles, making the edited image more realistic and natural.

[0236] During image editing, the user performs object movement and 3D rotation operations. At this time, the electronic device 200 can respond to the user's object movement and 3D rotation operations respectively, performing object translation editing and 3D rotation editing respectively. For example, see... Figure 12 The illustrated image editing operation flowchart provided in this application embodiment may include, but is not limited to, the following steps:

[0237] Step S1201: Input the 2D image to be edited.

[0238] After the user selects the image to be edited, the electronic device 200 can respond to the user's operation and identify the selected image as the image to be edited, thereby obtaining the 2D image to be edited. The 2D image to be edited is a two-dimensional image to be edited.

[0239] It is understood that step S1201 can be step 1, step a, and step S601 in the above embodiments. For details, please refer to the relevant content of step 1, step a, and step S601 in the above embodiments, which will not be repeated here.

[0240] Step S1202: Select a specific object in the image.

[0241] After acquiring the image to be edited, the electronic device 200 can determine the target object (i.e., a specific object) specified by the user in response to the user's object selection operation.

[0242] It is understood that step S1202 can be steps 2, b, and S602 in the above embodiments. For details, please refer to the relevant content of steps 2, b, and S602 in the above embodiments, which will not be repeated here.

[0243] Step S1203: Segment the objects in the diagram.

[0244] When selecting a target object, the user can do so through touch clicks, text input, or voice input. When the user selects a target object through touch clicks, the electronic device 200 can determine the selected target object based on the user's touch point (or selection point); when the user selects a target object through text input or voice input, the electronic device 200 can either convert the speech to text to obtain the text of the object to be edited, or directly obtain the text of the object to be edited, and thus determine the selected target object.

[0245] After determining the target object, the electronic device 200 can segment the object to be edited (i.e., the target object) from the image to be edited using an image semantic segmentation model.

[0246] After segmenting the target object from the image, the electronic device 200 can display the target object as selected.

[0247] Step S1204: Determine whether to rotate. If yes, proceed to step S1205; if no, proceed to step S1209.

[0248] After selecting a target object, the user can rotate or translate the target object in three dimensions. If the user rotates the target object in three dimensions, the electronic device 200 can proceed to steps S1205 to S1208; if the user translates the target object, the electronic device can proceed to steps S1210 to S1212.

[0249] Step S1205: Input the angle to be rotated.

[0250] If rotation is required, the user can input the desired rotation angle. The electronic device 200 can receive the rotation angle input by the user.

[0251] Step S1205 can be step c in the above embodiment. For details, please refer to the relevant content of step c above, which will not be repeated here.

[0252] Step S1206: Generate the viewpoint of the object to be edited after rotation.

[0253] Step S1207: Overlay the new object image onto the original image.

[0254] The original image can refer to the image obtained by removing the original object to be edited (i.e., the target object) from the 2D image to be edited.

[0255] Step S1208: Fill in the missing areas in the image.

[0256] After obtaining the rotation angle input by the user, the electronic device 200 generates the viewpoint of the target object after rotation, that is, generates the target object under the viewpoint after rotation, and superimposes the target object under the viewpoint after rotation onto the original image, and then fills the missing areas in the image.

[0257] Steps S1206 to S1208 can be step d in the above embodiment. For details, please refer to the relevant content of step c above, which will not be repeated here.

[0258] Step S1209: Determine whether to translate. If yes, proceed to step S1210.

[0259] If translation is required, the electronic device 200 selects the object and its shadow, responds to the user's drag-and-move operation, drags the object and its shadow to translate, overlays the object and its shadow onto the original image, and then fills in the missing areas of the image.

[0260] Step S1210: Select the object and its shadow.

[0261] The electronic device 200 can segment the objects and their shadows in the image to obtain the outer contours of the objects and their shadows; and use a graphic of a certain shape (e.g., a rectangle) to frame the outer contours based on the objects and their outer contours.

[0262] Step S1211: Drag the object and translate its shadow.

[0263] The electronic device 200 can respond to the user's dragging operation on the object and control the movement of the object and its shadow.

[0264] Step S1212: Translate the object image and overlay it on the original image, then proceed to step S1208 to fill the missing areas in the image.

[0265] The original image can refer to the image obtained by removing the original object to be edited (i.e., the target object) from the 2D image to be edited.

[0266] Steps S1210 to S1212 can be steps 3 and 4 in the above embodiments. For details, please refer to the relevant content of steps 3 and 4 above, which will not be repeated here.

[0267] It should be noted that when a user moves and rotates a target object in 3D, the movement and rotation operations can be performed step by step. For example, the movement operation can be performed first, followed by the 3D rotation operation. Of course, the movement and rotation operations can also be the same operation, meaning the user can move and rotate the target object simultaneously through a single operation.

[0268] Furthermore, when performing movement and 3D rotation step by step, the order between 3D rotation and movement can be set according to actual needs. The 3D rotation operation can be performed first, the movement operation can be performed first, or both can be performed simultaneously. No limitation is imposed here.

[0269] For example, see Figure 6C and Figure 6D While moving the striped cup 612, it can also be rotated in three dimensions. The viewing angle of the striped cup 612 is different before and after the movement. A comparison can be made to see the specifics. Figure 6C and Figure 6D The orientation of the spoon on the striped cup 612 is determined. During this process, the user can move and rotate the striped cup 612 in three dimensions. At this time, the electronic device 200 can also move and rotate the striped cup 612 in three dimensions according to the user's three-dimensional rotation and movement operations to generate an image 618. The three-dimensional operation and the movement operation can be the same operation or two operations, and the execution order of the two operations can be simultaneous or sequential.

[0270] Please see Figure 13 This is a schematic flowchart of an interactive image editing method provided in an embodiment of this application. The method may include the following steps:

[0271] Step S1301: Electronic device 200 displays an image to be edited, which includes at least one object.

[0272] For example, the electronic device 200 can display the image to be edited in the image editing interface after entering image editing mode. For example, in Figure 3B In the image editing interface 315 of the gallery application, the mobile phone 300 displays the image 307 to be edited.

[0273] In step S1302, the electronic device 200, in response to a first operation on at least one object, selects a target object from the at least one object, wherein the target object in the image to be edited is in a first position and the shadow of the target object is in a second position.

[0274] The first operation is used to select a target object from at least one object. The first operation can be a touch operation, a voice operation, or a text input operation, etc., and is not limited here.

[0275] In step S1303, the electronic device 200, in response to a second operation on the target object, displays a first image. The first image is obtained after moving the target object and its shadow in the image to be edited according to the second operation. The target object in the first image is at a third position, and the shadow of the target object is at a fourth position. The third position is the position indicated by the second operation. The second operation is used to move the target object; when the target object moves, its shadow moves along with the target object.

[0276] For example, the second operation is a drag-and-drop operation, that is, the user drags the target object to a specified position and then releases the target object to move the target object to that specified position.

[0277] For example, in Figure 3E In the diagram, the first position can refer to position A, the second position can refer to position B, the third position can refer to position C, and the fourth position can refer to position D.

[0278] As can be seen from the above, the embodiments of this application improve the image editing effect by making the shadow of the target object move with the movement of the target object.

[0279] In one optional embodiment, the electronic device 200 may first extract the target object and its shadow from the image to be edited; then superimpose the target object onto a third position of a second image, and make the shadow of the target object follow the target object to a fourth position of the second image to obtain a first image; and display the first image. The second image is the image obtained after removing the target object and its shadow from the image to be edited and filling in the first missing area, where the first missing area is the missing area that appears after removing the target object and its shadow.

[0280] In one alternative embodiment, the electronic device 200 can input a first mask of the image to be edited and the target object into a variational autoencoder to obtain the image tensor and mask tensor output by the variational autoencoder. Then, the image tensor and mask tensor are input into a shadow encoder to obtain a second mask output by the shadow encoder. The second mask is a mask of the target object and the shadow of the target object. Finally, the target object and the shadow of the target object are segmented from the image to be edited based on the second mask.

[0281] For example, see Figure 4C The first mask can be mask 361, and the second mask can be mask 362.

[0282] In an optional embodiment, the electronic device 200 can further determine the perspective scaling relationship and foreground / background occlusion relationship of the target object at the third position based on the depth information of the image to be edited. Based on the perspective scaling relationship, the target object and its shadow are scaled down or up. Then, based on the foreground / background occlusion relationship, the target object is superimposed onto the third position of the second image, so that the shadow of the target object follows the superimposed target object to the fourth position of the second image, thus obtaining the first image. The foreground / background occlusion relationship is used to characterize the occlusion relationship between the target object, its shadow, and other objects.

[0283] The perspective scaling relationship can be determined based on the difference in depth of field before and after the movement.

[0284] In an optional embodiment, the electronic device 200 can also adjust the hue of the target object and its shadow based on the hue of the third position when the hue of the target object and its shadow are different before and after the target object moves; and fill in the missing elements of the target object when there are missing elements.

[0285] In one optional embodiment, the electronic device 200 can also highlight the associated plane of the target object, and when the target object moves beyond the associated plane, a new associated plane of the target object is regenerated based on the current position of the target object and the new associated plane is highlighted; when the target object moves within the associated plane, the target object and its shadow can be moved to a specified position.

[0286] In one alternative embodiment, the image to be edited is a two-dimensional image with a single viewpoint, and the target object is in a first pose within the image to be edited. The electronic device 200 can also display a third image in response to a third operation on the target object. The third image is obtained by performing a three-dimensional rotation on the target object in the image to be edited according to the third operation. In the third image, the target object is in a second pose, and the third operation is used to perform a three-dimensional rotation on the target object from the first pose to the second pose.

[0287] For example, see Figure 10A and Figure 10B The third operation can be the user dragging the sliding control 342 on the rotation angle bar 341 to input the rotation angle. The first pose can refer to the pose shown by object 327 in image 308, and the second pose can be the pose after rotating 30°, that is, the pose of object 327 in image 349.

[0288] In one alternative embodiment, the electronic device 200 can segment the target object in a first pose from the image to be edited to obtain a fourth image, which is the image obtained after removing the target object from the image to be edited; fill in the second missing region in the fourth image to obtain a fifth image, where the second missing region is the missing region that appears after removing the target object; generate the target object in a second pose based on the target object in the first pose, the pose of the third operation instruction, and the rotation task; superimpose the target object in the second pose and the fifth image to obtain a third image; and display the third image.

[0289] For example, see Figure 10A and Figure 10B The fourth image can refer to image 346, the fifth image can refer to image 348, the second missing region can refer to missing region 347, and the third image can refer to image 349.

[0290] In one alternative embodiment, the electronic device 200 can generate an input image based on the target object in the first pose and a third mask of the target object; generate an embedding vector based on the pose of the first pose, the pose of the third operation instruction, and the rotation task; input the input image and the embedding vector into a generation model to obtain the target object in the second pose output by the generation model; the embedding vector is the key and value input of the generation model, and the input image is the query input of the generation model. For example, this process can be as follows: Figures 10C to 10E As shown.

[0291] In an optional embodiment, the electronic device 200 can further remove the shadow of the target object in the first pose in the fourth image, that is, remove the shadow of the target object in the original pose in the image. At this time, in the process of superimposing the target object in the second pose and the fifth image to obtain the third image, the electronic device can generate the shadow of the target object in the second pose; superimpose the target object in the second pose and the shadow of the target object in the second pose on the fifth image to obtain the third image.

[0292] In one alternative embodiment, the image editing interface displays a rotation angle adjustment control, which is used to adjust the three-dimensional rotation angle of the target object; the third operation is an input operation on the rotation angle control.

[0293] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0294] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0295] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0296] In the embodiments provided in this application, it should be understood that the disclosed devices, electronic devices, and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0297] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0298] The electronic device provided in this application embodiment may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method as described in any of the above method embodiments.

[0299] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0300] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0301] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the methods described in the above embodiments. The chip system may be a single chip or a chip module composed of multiple chips.

[0302] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or recorded in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, in the description of this application specification and appended claims, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," "third," and "fourth" may explicitly or implicitly include one or more of that feature. Additionally, it should be understood that at least one in the embodiments of this application includes one or more; where "more" means greater than or equal to two. In the embodiments of this application, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the objects before and after it are in an "or" relationship.

[0303] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in still other embodiments" appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0304] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interactive image editing method, characterized in that, Applied to electronic devices, the method includes: Display an image to be edited, the image comprising at least one object; In response to a first operation on the at least one object, a target object is selected from the at least one object, wherein the target object in the image to be edited is in a first position and the shadow of the target object is in a second position; In response to a second operation on the target object, a first image is displayed. The first image is obtained by moving the target object and its shadow in the image to be edited according to the second operation. The target object in the first image is in a third position, and the shadow of the target object is in a fourth position. The third position is the position indicated by the second operation. The second operation is used to move the target object; as the target object moves, the shadow of the target object moves with the target object.

2. The method according to claim 1, characterized in that, In response to a second operation on the target object, displaying a first image includes: Extract the target object and its shadow from the image to be edited; The target object is superimposed on the third position of the second image, and the shadow of the target object is superimposed on the fourth position of the second image along with the target object to obtain the first image. The second image is the image obtained after removing the target object and the shadow of the target object from the image to be edited and filling in the first missing area. The first missing area is the missing area that appears after removing the target object and the shadow of the target object. Display the first image.

3. The method according to claim 2, characterized in that, Extracting the target object and its shadow from the image to be edited includes: The first mask of the image to be edited and the target object are input into the variational autoencoder to obtain the image tensor and mask tensor output by the variational autoencoder; The image tensor and the mask tensor are input into the shadow encoder to obtain the second mask output by the shadow encoder. The second mask is a mask of the target object and the shadow of the target object. Based on the second mask, the target object and its shadow are segmented from the image to be edited.

4. The method according to claim 2, characterized in that, The first image is obtained by superimposing the target object onto the third position of the second image, and causing the shadow of the target object to follow the superimposition onto the fourth position of the second image, including: Based on the depth information of the image to be edited, determine the perspective scaling relationship and foreground / background occlusion relationship of the target object at the third position; Based on the perspective scaling relationship, after shrinking or enlarging the target object and its shadow, based on the foreground and background occlusion relationship, the target object is superimposed onto the third position of the second image, so that the shadow of the target object follows the target object and is superimposed onto the fourth position of the second image, thus obtaining the first image.

5. The method according to claim 4, characterized in that, Before overlaying the target object onto the third position of the second image and causing the shadow of the target object to follow the target object overlay onto the fourth position of the second image, the method further includes: When the hue of the target object and its shadow are different before and after the target object moves, the hue of the target object and its shadow is adjusted according to the hue of the third position. If the target object is missing, the target object is padded with elements.

6. The method according to claim 1, characterized in that, The method further includes: Highlight the associated plane of the target object; During the movement of the target object, when the movement range of the target object exceeds the associated plane, a new associated plane of the target object is regenerated based on the current position of the target object, and the new associated plane is highlighted.

7. The method according to any one of claims 1 to 6, characterized in that, The image to be edited is a two-dimensional image from a single perspective, and the target object is in the first pose of the image to be edited; The method further includes: In response to a third operation on the target object, a third image is displayed. The third image is an image obtained by performing a three-dimensional rotation on the target object in the image to be edited according to the third operation. In the third image, the target object is in a second pose. The third operation is used to perform a three-dimensional rotation on the target object to rotate it from the first pose to the second pose.

8. The method according to claim 7, characterized in that, In response to a third operation on the target object, displaying a third image includes: The target object in the first pose is segmented from the image to be edited to obtain a fourth image, wherein the fourth image is the image obtained after removing the target object from the image to be edited; The second missing region in the fourth image is filled in to obtain the fifth image, where the second missing region is the missing region that appears after the target object is removed. The target object of the second pose is generated based on the target object of the first pose, the pose of the third operation instruction, and the rotation task; The third image is obtained by superimposing the target object in the second pose and the fifth image; The third image is displayed.

9. The method according to claim 8, characterized in that, Based on the target object of the first pose, the pose of the third operation instruction, and the rotation task, the target object of the second pose is generated, including: An input image is generated based on the target object in the first pose and the third mask of the target object; An embedding vector is generated based on the first pose, the pose of the third operation instruction, and the rotation task; The input image and the embedding vector are input into the generative model to obtain the target object of the second pose output by the generative model; the embedding vector is the key and value input of the generative model, and the input image is the query input of the generative model.

10. The method according to claim 8, characterized in that, The method further includes: Remove the shadow of the target object in the first pose from the fourth image; The third image is obtained by superimposing the target object in the second pose and the fifth image, including: Generate the shadow of the target object in the second pose; The target object in the second pose and its shadow are superimposed on the fifth image to obtain the third image.

11. The method according to claim 7, characterized in that, The image editing interface displays a rotation angle adjustment control, which is used to adjust the three-dimensional rotation angle of the target object; The third operation is an input operation for the rotation angle adjustment control.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 11.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

Images