Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method and a related device, and relates to the technical field of terminals. The method comprises the steps that a plurality of first images are shot in a first mode, a second image is shot in a second mode, then a target image is obtained based on the first images and the second image, and the definition of the image shot in the first mode is higher than that of the image shot in the second mode. And the field angle of the image shot in the first mode is lower than the field angle of the image shot in the second mode, so that the target image can be obtained by combining the advantages of high field angle and high definition, the technical problem of poor definition of the object in the group photo image in the related technology is solved, and the image quality is improved. The technical effect of improving the object definition in the group photo image is achieved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] In daily life, people often use electronic devices to shoot multiple subjects, such as group photos of multiple people. Usually, for scenes with a large number of people, the rear main camera or ultra-wide mode is selected to shoot to ensure that everyone is within the frame.

[0003] However, the clarity of the characters in the group photo images needs to be further improved. Summary of the invention

[0004] The embodiment of the present application provides an image processing method and related devices, which are applied to the field of terminal technology. The embodiment of the present application enhances the target image in the second image by using multiple first images with higher clarity and lower field of view than the second image, thereby solving the technical problem of poor clarity of objects in group photo images in the related art and achieving the technical effect of improving the clarity of objects in group photo images.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes:

[0006] A first interface is displayed, the first interface includes a first preview image and a first button; in response to a first operation on the first button, prompt information and a second button are displayed, the prompt information is used to prompt the acquisition of images of multiple target objects; within a target time period, multiple first images are captured using the first mode, and the multiple first images are displayed respectively; in response to a second operation on the second button, a second image is captured using the second mode, wherein the second button is continuously displayed within the target time period, the target time period includes a time period from receiving the first operation to receiving the second operation, the clarity of the image captured using the first mode is higher than the clarity of the image captured using the second mode, and the field of view of the image captured using the first mode is lower than the field of view of the image captured using the second mode; a target object in the second image is enhanced using a stitched image to obtain a target image, and the stitched image includes an image obtained by stitching multiple first images.

[0007] The first interface may be an interface for taking a group photo. Optionally, the first interface may be an interface for entering a group photo mode. The first button may be a button for triggering the electronic device to take an image in the first mode.

[0008] Exemplarily, the first interface may be as follows Figure 5A or Figure 5B In the interface shown in FIG. 1 , the first button may be as follows: Figure 5Aor Figure 5B the shutter button.

[0009] The second button may be a button for triggering the end of capturing the first image.

[0010] The target time period may refer to the time period for capturing the first image. It should be noted that the first image captured within the target time period only means that the first image is captured within the time period between receiving the user operation of the first button and receiving the user operation of the second button, and does not mean that the electronic device is actually timing.

[0011] Exemplarily, the multiple first images of this embodiment may be as shown in FIG. Figure 6B The image displayed in the preview window 201 shown in FIG. Figure 6C-6D The preview window 201 shown displays an image.

[0012] Optionally, each first image may include part of the target objects among all the objects, the second image may include all the target objects, the target image includes all the target objects, and the clarity of each target object in the target image is higher than the clarity of the target object in the second image.

[0013] In an embodiment of the present application, the target image in the second image is enhanced by using multiple first images with higher clarity and lower field of view than the second image, thereby solving the technical problem of poor clarity of objects in group photos in the related art and achieving the technical effect of improving the clarity of objects in group photos.

[0014] In conjunction with the first aspect, in a possible implementation manner, the prompt information includes information for prompting the user to keep the posture of the electronic device unchanged, and the plurality of first images are captured by the electronic device through a movable camera;

[0015] Alternatively, the prompt information includes information for guiding the movement trajectory, and the multiple first images are taken by the mobile electronic device.

[0016] In this embodiment, the specific prompt information may be related to the way in which the electronic device captures the first image. For example, the prompt information of the electronic device controlling the camera to move and capturing the first image during the camera movement may be different from the prompt information of the user moving the electronic device to capture the first image during the movement.

[0017] Exemplarily, the second button may be Figure 6A-6E The "Finish button" shown in the figure can have the following prompt information: Figure 6A-6B The first prompt message shown, or Figure 6C-6D The second prompt information shown is not limited here.

[0018] It can be understood that by prompting the user to keep the posture of the electronic device unchanged and capturing multiple first images in the process of controlling the movement of the camera by the electronic device, the user does not need to move the electronic device, which can reduce the user's operating steps and improve the intelligence and convenience of the electronic device's shooting.

[0019] In addition, the moving trajectory of the electronic device is guided by prompt information, and the user moves the electronic device to capture multiple first images. In this way, there is no need to set a device for controlling the movement of the camera in the electronic device, so the structure of the electronic device can be simplified.

[0020] In combination with the first aspect, in a possible implementation manner, the image processing method further includes:

[0021] Displaying a second interface, the second interface includes a second preview image, a third button and a fourth button, the third button is in an unselected state, the fourth button is in a selected state, and the second preview image is a preview image captured in the mode of the fourth button;

[0022] The first interface is displayed, including:

[0023] In response to the third operation on the third button, a first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

[0024] The second interface may refer to an interface displayed when the camera application is opened. The third button may refer to a button that triggers the display of the first interface.

[0025] Exemplarily, the second interface may be as follows Figure 4A In the interface shown in FIG. 1 , the third button can be as follows: Figure 4A The photo mode option 2021 shown, the fourth button can be as follows Figure 4A Photo mode options shown.

[0026] In an embodiment of the present application, the third button can be directly displayed on the second interface, so that when the third operation on the third button is detected, the first interface is entered, and then multiple first images can be captured on the first interface. In this way, the efficiency of capturing multiple first images can be improved, thereby improving the efficiency of image enhancement.

[0027] In combination with the first aspect, in a possible implementation manner, the image processing method further includes:

[0028] Displaying a third interface, the third interface includes a second preview image, a fourth button and a fifth button, the fourth button is in a selected state, the second preview image is a preview image captured in the mode of the fourth button, and the fifth button is in an unselected state;

[0029] In response to a fourth operation on the fifth button, a fourth interface is displayed, where the fourth interface includes a third button, and the third button is in an unselected state;

[0030] The first interface is displayed, including:

[0031] In response to the third operation on the third button, a first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

[0032] Among them, the third interface may refer to the interface displayed when the camera application is opened, and the fifth button may refer to the button that triggers the display of the fourth interface.

[0033] Exemplarily, the third interface may be as follows Figure 4B The fourth button can be as follows Figure 4B The fourth interface can be as shown in the shooting mode options, the fifth button can be a more button. Figure 4C As shown, the third button may refer to Figure 4C Group photo mode options 2021 shown.

[0034] For example, the display of the second interface or the third interface may be that the electronic device detects that the Figure 3 When the camera application icon 101 is shown, the interface displayed on the display screen of the electronic device.

[0035] In combination with the first aspect, in a possible implementation manner, using the stitched image to enhance the target object in the second image to obtain the target image includes:

[0036] The second image is segmented to obtain an object region mask; the spliced ​​image is corrected based on the object region mask to obtain a reference image, wherein the size of the object region in the reference image matches the size of the object region mask; the second image is super-resolved based on the reference image and the object region mask to obtain a target image.

[0037] The object region may refer to a region that needs to be enhanced. For example, if the second image is a group photo of people, the object region may be a portrait region. For another example, if the second image is an image of multiple plants, the object region may be a plant. The size of the object region in the reference image matches the size of the object region mask, which may mean that the size of the object region in the reference image is consistent with the size of the object region mask.

[0038] The target image can be understood as a second image with higher definition. Specifically, the definition of the object in the target image is higher than the definition of the object in the second image.

[0039] In an embodiment of the present application, the second image is segmented to obtain an object area mask; the spliced ​​image is corrected based on the object area mask to obtain a reference image, and the size of the object area in the reference image matches the size of the object area mask; the second image is super-resolved based on the reference image and the object area mask to obtain a target image, that is, the spliced ​​image is first corrected to obtain the reference image, and then super-resolved based on the reference image, so that the reference image is more closely matched with the object area in the second image than the spliced ​​image, and thus, the accuracy and efficiency of super-resolution based on the reference image can be higher than that of super-resolution on the spliced ​​image.

[0040] In combination with the first aspect, in a possible implementation manner, super-resolving the second image based on the reference image and the object region mask to obtain the target image includes:

[0041] Feature extraction is performed on the reference image and the second image respectively to obtain reference image features of the reference image and second image features of the second image; feature alignment is performed on the reference image features and the second image features based on the object area mask to obtain a pixel-level offset map, and the pixel-level offset map indicates the offset information of each pixel in the second image in the reference image; the reference image is encoded (encoder) to obtain a reference feature map; a deformable convolution operation or an attention-based convolution operation is performed on the reference feature map based on the pixel-level offset map to obtain texture features to be migrated; the texture features to be migrated are migrated to the second image to obtain a target image.

[0042] Among them, texture features can refer to regularly arranged patterns within a certain range in the image. Texture is one of the important features of image processing.

[0043] In an embodiment of the present application, the offset direction and amount of each pixel of the second image in the reference image can be determined based on the pixel-level offset map. In this way, the variable convolution operation or the attention-based convolution operation on the encoded feature map takes into account the pixel offset between the reference image and the second image. In this way, the correlation between the obtained migrated texture features and the object area is higher, and the enhanced area of ​​the target image obtained after migration to the second image is more accurate.

[0044] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0045] Performing single-frame super-resolution on the second image to obtain a second image after single-frame super-resolution;

[0046] Migrating the texture features to be migrated to the second image to obtain a target image, including:

[0047] The texture features to be migrated are migrated to the second image after single-frame super-resolution to obtain the target image.

[0048] The single-frame super-resolution may be super-resolution of the entire second image, including but not limited to super-resolution of the object area and super-resolution of the non-object area. Taking a group portrait as an example, super-resolution may be performed on the portrait area and the background area.

[0049] In the embodiment of the present application, the second image is subjected to single-frame super-resolution to obtain the second image after single-frame super-resolution, and then the texture features to be migrated are migrated to the second image after single-frame super-resolution to obtain the target image, that is, at least two levels of image super-resolution are performed, so that the super-resolution effect of the object area can be further improved, that is, the clarity of the object area can be improved. In addition, the non-object area can also be super-resolved, so that the overall clarity of the image is also improved.

[0050] In combination with the first aspect, in a possible implementation method, a target image is obtained by super-resolution based on a reference image, an object region mask and a second image through a pre-trained super-resolution model, and the pre-trained super-resolution model includes a feature extraction network, a feature alignment network, an encoder, a convolutional network and a texture transfer network; the feature extraction network is used to extract features from the reference image and the second image respectively to obtain reference image features of the reference image and second image features of the second image; the feature alignment network is used to align the reference image features and the second image features based on the object region mask to obtain a pixel-level offset map; the encoder is used to encode the reference image to obtain a reference feature map; the convolutional network is used to perform a deformable convolution operation or an attention-based convolution operation on the reference feature map based on the pixel-level offset map to obtain texture features to be transferred; the texture transfer network is used to migrate the texture features to be transferred to the second image to obtain the target image.

[0051] In combination with the first aspect, in a possible implementation manner, the target image is obtained by migrating the texture features to be migrated to a second image after single-frame super-resolution, and the pre-trained super-resolution model further includes a single-frame super-resolution network;

[0052] The single-frame super-resolution network is used to perform single-frame super-resolution on the second image to obtain a second image after single-frame super-resolution;

[0053] The texture transfer network is used to transfer the texture features to be transferred to the second image after single-frame super-resolution to obtain the target image.

[0054] In combination with the first aspect, in a possible implementation manner, correcting the spliced ​​image based on the object region mask to obtain a reference image includes:

[0055] The stitched image is stitched with the object area mask, and features are extracted from the stitched image with the object area mask stitched thereon to obtain a stitched feature map; correction processing is performed N times based on the stitched feature map to obtain a reference image, where N is an integer not less than 2;

[0056] Each calibration process includes:

[0057] Perform network regression on the input, as well as mesh motion and bending processing to obtain the output;

[0058] Among them, the input of the first correction process is the spliced ​​image with the object area mask spliced, the input of the Mth correction process is the output obtained by the M-1th correction process, and the output obtained by the Nth correction process is used as the reference image, 1<M≤N, and M is an integer.

[0059] In the embodiment of the present application, after N correction processes, that is, multi-level correction processes, rough correction processes and refined correction processes can be performed in sequence, so that the effect of the correction process can be improved, and then the effect of image enhancement can be improved.

[0060] In a second aspect, the present application also provides an image processing method, including:

[0061] A second image and multiple first images are acquired, wherein the multiple first images are acquired by shooting in the first mode, and the second image is acquired by shooting in the second mode, wherein the clarity of the image acquired by shooting in the first mode is higher than the clarity of the image acquired by shooting in the second mode, and the field of view of the image acquired by shooting in the first mode is lower than the field of view of the image acquired by shooting in the second mode; the multiple first images are stitched to acquire a stitched image; and a target object in the second image is enhanced by using the stitched image to acquire a target image.

[0062] In an embodiment of the present application, the image processing method may also be executed by a server. For example, after the electronic device captures the first image and the second image, the electronic device sends the first image and the second image to the server, and the server returns the target image based on the first image and the second image, and the electronic device stores the target image.

[0063] In a third aspect, the present application also provides a method for training a super-resolution model, the method comprising:

[0064] A super-resolution model to be trained and training samples are obtained. The super-resolution model to be trained includes a feature extraction network, a feature alignment network, an encoder, a convolutional network and a decoder. The feature extraction network is used to extract features from a reference image and a second image respectively to obtain reference image features of the reference image and second image features of the second image. The feature alignment network is used to align the reference image features and the second image features based on an object region mask to obtain a pixel-level offset map. The pixel-level offset map indicates offset information of each pixel in the second image in the reference image. The object region mask is obtained by segmenting the second image. The encoder is used to encode the reference image to obtain a reference feature map. The convolutional network is used to perform a deformable convolution operation or an attention-based convolution operation on the reference feature map based on the pixel-level offset map to obtain texture features to be transferred. The decoder is used to decode the texture features to obtain texture information. The texture information is used to supervise the training process of the super-resolution model to be trained. The super-resolution model to be trained is trained based on the training samples until the training of the super-resolution model to be trained is completed to obtain a pre-trained super-resolution model.

[0065] In an embodiment of the present application, after the super-resolution model is trained, image enhancement processing can be performed based on the trained super-resolution model, so that the clarity of the object in the image can be improved.

[0066] In a fourth aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The image processing device may include a display unit and a processing unit. When the image processing device is an electronic device, the display unit may be a display screen. The display unit is used to perform the display step so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is an electronic device, the processing unit may be a processor. The image processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0067] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, or any possible implementation of the third aspect.

[0068] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, or any possible implementation of the third aspect.

[0069] In the seventh aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, or any possible implementation of the third aspect.

[0070] In an eighth aspect, the present application provides a chip or a chip system, the chip or chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect, the second aspect, the third aspect, any possible implementation of the first aspect, any possible implementation of the second aspect, or any possible implementation of the third aspect. Among them, the communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0071] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0072] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of an interface for opening a camera application provided in an embodiment of the present application;

[0076] Figure 4A-4C A schematic diagram showing a default shooting preview interface provided in an embodiment of the present application;

[0077] Figure 5A-Figure 5B A schematic diagram of an interface for entering a group photo mode provided in an embodiment of the present application;

[0078] Figure 6A-6E A schematic diagram of an interface for shooting multiple first images provided in an embodiment of the present application;

[0079] Figure 7 A schematic diagram of an interface for capturing a second image provided in an embodiment of the present application;

[0080] Figure 8 A schematic diagram of a specific implementation of an image processing method provided in an embodiment of the present application;

[0081] Fig. 9 A schematic diagram of a specific implementation of image enhancement provided in an embodiment of the present application;

[0082] Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0084] 1. Super resolution (SR), also known as super resolution for short. The opposite of super resolution is low resolution image (LR). Super resolution is a low-resolution image processing task that maps a low-resolution image to a high resolution in order to enhance image details. There are many reasons why an image is blurry, such as various types of noise, lossy compression, downsampling, and so on. Super resolution is a classic application of computer vision. SR refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image through software or hardware methods (in other words, improving the resolution), which has important application value in monitoring equipment, satellite image remote sensing, digital high-definition, microscopic imaging, video coding communication, video restoration, and medical imaging.

[0085] 2. The field of view angle is also called the field of view in optical engineering. The size of the field of view angle determines the field of view of the optical instrument. The field of view angle can also be expressed as FOV, and its relationship with the focal length is as follows: image height = EFL*tan(half FOV); EFL is the focal length; FOV is the field of view angle. The size of the field of view angle determines the field of view of the optical instrument. The larger the field of view angle, the larger the field of view and the smaller the optical magnification. In layman's terms, the target object will not be included in the lens if it exceeds this angle. According to the field of view angle classification, it includes but is not limited to the following types of lenses:

[0086] Standard lens: viewing angle is about 45 degrees, with a wide range of use.

[0087] Telephoto lens: The viewing angle is within 40 degrees and can be used to shoot at long distances.

[0088] Wide-angle lens: The viewing angle is more than 60 degrees, the observation range is larger, and the image is distorted at close range.

[0089] 3. Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing targets of interest. It is a key step from image processing to image analysis. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-intersecting regions. The image segmentation process is also a labeling process, that is, pixels belonging to the same region are given the same number.

[0090] In an embodiment of the present application, the second image can be segmented to obtain an object area mask. A mask can be understood as data similar to an image. In an embodiment of the present application, the image and the mask can be fused to make part of the content in the image more noticeable. Generally, a mask can be used to extract an area of ​​interest, such as fusing a pre-made area of ​​interest mask with an image to be processed to obtain an area of ​​interest image, where the image values ​​in the area of ​​interest remain unchanged, while the image values ​​outside the area are all 0. It can also play a shielding role, using a mask to shield certain areas on the image so that they do not participate in the processing or the calculation of processing parameters, or only process or count the shielded area, etc. In an embodiment of the present application, it can be a mask of an object area, such as a mask of a portrait area or a mask of a body area.

[0091] 4. Feature map refers to the image features extracted from the input image through convolution operation in convolutional neural network. In convolutional neural network, the output of each layer is a three-dimensional tensor, where the third dimension represents the number of feature maps. Each feature map is obtained by convolving the feature map of the previous layer with several convolution kernels, and each convolution kernel corresponds to a specific feature. Therefore, the feature map can be regarded as the response result to certain specific features in the input image, which can be used to understand the working principle of convolutional neural network and visualize the feature extraction process of convolutional neural network.

[0092] 5. Image features Image features mainly include: color features, shape features, texture features and spatial relationship features.

[0093] 6. The purpose of image stitching rectangularization is to solve the problem of irregular boundaries after image stitching. Existing image stitching rectangularization methods are usually divided into two stages: the first stage is to search for an initial grid, that is, to place a regular grid on the stitched image to describe the position of each point on the image; the second stage is to optimize a target grid, that is, to perform some deformation on the basis of the initial grid so that the grid edge is aligned with the rectangular boundary as much as possible. Then, by transforming the stitched image from the initial grid to the target grid, a rectangular image can be obtained. This process is called grid deformation or grid distortion.

[0094] 7. The specific implementation of texture migration super-resolution processing can be in many ways, for example, by implementing texture migration super-resolution processing through a neural network model (texture migration super-resolution neural network model) to output a super-resolution video. The neural network model can be any neural network model, for example, a deep neural network (DNN), a convolutional neural network (CNN) or a combination thereof.

[0095] 8. Other terms

[0096] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0097] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0098] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0099] 9. Electronic devices

[0100] The electronic devices of the embodiments of the present application may include handheld devices, vehicle-mounted devices, etc. with ** functions. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (public landmobile networks). The embodiments of the present application do not limit this.

[0101] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0102] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0103] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0104] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0105] The following embodiments illustrate the scenarios of the embodiments of the present application.

[0106] In some example situations, such as Figure 1 As shown, a group photo needs to be taken. Generally, in order to make the image capture able to accommodate all people, it is necessary to use the rear main camera or ultra-wide-angle mode to shoot, so that the field of view of the shot is large enough to accommodate all people. However, a large field of view generally means a short focal length, and a short focal length means that a certain degree of image clarity needs to be sacrificed, which will result in poor clarity of the people in the group photo.

[0107] In other exemplary situations, the focal length of the telephoto lens is longer than that of the main camera or ultra-wide-angle camera, and the image taken with the telephoto lens has higher clarity than that taken with the main camera or ultra-wide-angle camera, but the field of view of the telephoto lens is also smaller than that of the main camera or ultra-wide-angle camera, and the image taken with the telephoto lens may not be able to accommodate everyone.

[0108] In view of this, an embodiment of the present application proposes an image processing method and a related device, which can enhance the second image through multiple first images. Since the first image has higher clarity but a smaller field of view than the second image, the advantages of the field of view of the first image and the clarity of the second image can be combined to obtain a target image with a large field of view and high clarity. In this way, compared with the second image, the clarity of the object in the target image is higher, thereby solving the technical problem of poor clarity of objects in group photos in the related technology and achieving the technical effect of improving the clarity of objects in group photos.

[0109] In order to better understand the embodiments of the present application, the structure of the electronic device in the embodiments of the present application is introduced below:

[0110] Figure 1 A schematic diagram of the hardware structure of the electronic device 10 is shown.

[0111] The electronic device 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0112] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0113] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0114] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0115] In the embodiment of the present application, the camera 193 can collect preview image data, collect first image data and second image data, and display the image on the display screen 194. The camera 193 may include a main camera for collecting preview image data, a telephoto camera for collecting first image data, and a main camera or an ultra-wide-angle camera for collecting second image data. The processor 110 may perform image enhancement processing, for example, by using a neural network processor to perform image enhancement processing based on a neural network model.

[0116] Figure 2 A schematic diagram of the software structure of the electronic device 10 is shown.

[0117] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0118] Figure 2 1 is a software structure diagram of the electronic device 100 of an embodiment of the present invention. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom, respectively, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0119] The application layer can include a series of application packages.

[0120] like Figure 2As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0121] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0122] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0123] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0124] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0125] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0126] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0127] The hardware abstraction layer is an interface layer between the operating system kernel and the upper software, and its purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface driven by the device kernel, which is used to provide the application programming interface for accessing the underlying device to the higher-level Java API framework. HAL contains multiple library modules, such as camera HAL, display HAL, etc.

[0128] Each library module implements an interface for a specific type of hardware component. It can be understood that the camera HAL can provide the camera FWK with an interface to access hardware components such as the camera. The display HAL can provide the display FWK with an interface to access hardware components such as the display. When the system framework layer API requires access to the hardware of the portable device, the Android operating system will load the library module for the hardware component.

[0129] The kernel layer is a layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver. For example, the camera driver is used to control the camera to collect image data. Optionally, the camera driver may include a main camera driver, an ultra-wide-angle camera driver, and a telephoto camera driver.

[0130] The following is an exemplary description of the workflow of electronic device software.

[0131] For example, when the touch sensor in the terminal device receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, touch strength, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the button corresponding to the input event. Taking the touch operation as a click operation, and the virtual button corresponding to the click operation as an "image capture button" as an example, the camera application calls the interface of the application framework layer, and then starts the display driver and the telephoto camera driver by calling the kernel layer, and the display screen displays the first image captured by the telephoto camera. Then, click the "Finish" virtual button, and the main camera driver or the ultra-wide-angle camera driver is started by calling the kernel layer, and the display screen displays the second image captured by the main camera or the ultra-wide-angle camera. Then, the camera application generates a target image based on the first image and the second image, and saves the target image in an image storage file, such as in an "album".

[0132] The following describes in detail the shooting scenarios provided by the present application in conjunction with some exemplary user interface diagrams.

[0133] It is to be understood that the terms "interface" and "user interface" in the embodiments of the present application are media interfaces for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, button, etc. displayed on a display screen of an electronic device, wherein a button may include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0134] 1. Open the Camera app ( Figure 3 ).

[0135] like Figure 3As shown, the user interface 100 (also called the main interface) displays a page with application icons, which may include multiple application icons (e.g., camera application, weather application icon, calendar application icon, album application icon, note application icon, email application icon, application store application icon, setting application icon, etc.). A page indicator may also be displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (e.g., camera application icon 101, browser application icon, information application icon, dial application icon) below the page indicator. These application icons remain displayed when the page is switched.

[0136] It can be understood that the camera application icon 101 is an icon of a camera application (ie, a camera application). The camera application icon 101 can be used to trigger the launch of the camera application.

[0137] In this embodiment, the electronic device may detect a user operation on the camera application icon 101 , and in response to the operation, the electronic device may display an initial shooting preview interface.

[0138] It is understandable that the user operations mentioned in this application may include but are not limited to touch (for example, click, etc.), voice control, gestures and other operations, and this application does not limit this.

[0139] 2. Display the default shooting preview interface ( Figure 4A-4C ).

[0140] (I) The default shooting preview interface includes the group photo mode option ( Figure 4A ).

[0141] like Figure 4A As shown, the user interface 200 is a shooting interface of the default shooting mode of the camera application, and the user can preview the image and complete the shooting on this interface.

[0142] The user interface 200 may include a preview window 201 , a camera mode option 202 , an album shortcut button, a shutter button (also called a capture button), and a camera flip button.

[0143] The preview window 201 of the user interface 200 can be used to display a preview image. The preview image displayed in the preview window 201 of the user interface 200 can be an image captured by the camera of the electronic device based on the framing range when the photo mode option is selected.

[0144] One or more shooting mode options may be displayed in the camera mode option 202. The one or more shooting mode options may include: an aperture mode option, a night scene mode option, a smart portrait mode option, a photo mode option, a group photo mode option 2021, a movie mode option, and more options.

[0145] It is understandable that the camera mode option 202 may also include more or fewer shooting mode options.

[0146] (ii) The default shooting preview interface does not include the group photo mode option ( Figure 4B-4C ).

[0147] like Figure 4B As shown, user interface 200 does not include group photo mode option 2021.

[0148] In such Figure 4B In the user interface 200 shown, the electronic device can detect the Figure 4B In response to a user operation of a more option button, the electronic device may display Figure 4C The user interface 300 shown. In the user interface 300, photo mode options other than the photo mode options displayed in the user interface 200 may be displayed, for example, the user interface 300 displays a professional mode option, a panoramic mode option, a high-dynamic range (HDR) mode option, a time-lapse photography mode option, a watermark mode option, a document correction mode option, a high-pixel mode option, and a group photo mode option 2021.

[0149] It is understandable that, in the user interface 300, in addition to the group photo mode option 2021, other mode options may be displayed according to actual conditions.

[0150] It should be noted that in the user interface 200, when the electronic device detects a sliding operation (such as a left or right slide) on the camera mode option 202, the camera mode option displayed by the camera mode option 202 is updated, thereby displaying the group photo mode option 2021.

[0151] When the electronic device detects a user operation on the group photo mode option 2021, the electronic device enters the group photo mode.

[0152] 3. Enter group photo mode ( Figure 5A-Figure 5B )

[0153] (I) After entering the group photo mode, a preview image with low field of view and high definition is displayed. Figure 5A )

[0154] like Figure 5A As shown, the preview image displayed in the preview window of the user interface 400 may be an image captured by the camera of the electronic device based on the framing range when the group photo mode option 2021 is selected. Multiple characters may be displayed in the preview image.

[0155] For example, Figure 5AAs shown, after entering the group photo mode, the preview image displayed in the preview window can be of low field angle and high definition.

[0156] (ii) After entering the group photo mode, a preview image with a high field of view and low resolution is displayed ( Figure 5B ).

[0157] like Figure 5B As shown, the preview image displayed in the preview window of the user interface 400 may be an image with a high field of view and low definition.

[0158] When the electronic device detects a user operation on the shutter button, the electronic device starts to capture multiple images with a low field of view and high definition.

[0159] 4. Take multiple images with low field of view and high resolution ( Figure 6A-6E ).

[0160] (i) Electronic equipment automatically captures multiple images with low field of view and high resolution ( Fig. 6A , Figure 6B and Fig. 6E ).

[0161] like Fig. 6A , Figure 6B and Fig. 6E As shown, when the electronic device detects a user operation on the shutter button, the camera of the electronic device starts to shoot and save multiple images with low field angle and high definition. During the process of the camera shooting multiple images with low field angle and high definition, the preview image displayed in the preview window 201 can also change in real time, so that the user can determine whether the multiple images with low field angle and high definition include all the people. During the shooting process, a text prompt message such as "All the people in the group photo are in the frame, please keep the phone fixed during the shooting process" can be continuously displayed.

[0162] It is understandable that the prompt information may be used to require that the content of the portrait group photo area in the image with a high field angle and low definition has been fully included in the multiple images with a low field angle and high definition in the previous step.

[0163] (ii) A user operates an electronic device to capture multiple images with low field of view and high resolution (6C- Fig. 6E )

[0164] like Figure 6C-6E As shown, optionally, a message such as "All people in the group photo are framed separately, please move your phone as estimated below" can be continuously displayed, and corresponding movement trajectory guidance can be given. Each time the electronic device detects a user operation on the shutter button, it takes a plurality of images with a low field of view and high definition and saves them.

[0165] It should be noted that the guidance of the moving trajectory is not limited to moving in a direction parallel to the horizontal line, but can also be moving in a direction perpendicular to the horizontal line, as long as all the people in the group photo can be included in the frame, and there is no limitation here.

[0166] When the electronic device detects a user operation on the completion button, the capturing of the image with a low field angle and high definition is stopped, and at this time, the capturing of the image with a high field angle and low definition can be started.

[0167] 5. Take images with high field of view and low resolution ( Figure 7 )

[0168] Exemplarily, the user interface 400 may display a prompt message to prompt that all the people need to be in the frame at the same time. For example, the prompt message may include text prompt messages such as "all the people in the group photo are in the frame at the same time, please keep the phone fixed during the shooting process".

[0169] When the electronic device detects a user operation on the shutter button, an image with a high field angle and low definition is captured.

[0170] It should be noted that the electronic device may also detect Fig. 6E When the user operates the Done button shown, an image with a high field angle and low definition is automatically captured.

[0171] In this way, multiple low-field-angle and high-definition images can be obtained, each low-field-angle and high-definition image includes some of the all characters, and the set of characters in all low-field-angle and high-definition images includes all the characters. At the same time, the high-field-angle and low-definition image includes all the characters. Furthermore, image enhancement can be performed based on the high-field-angle and low-definition image and the multiple low-field-angle and high-definition images to obtain the target image, so that the clarity of the characters in the obtained target image is higher than the clarity of the characters in the high-field-angle and low-definition images.

[0172] Then, the target image is saved in the album. When the electronic device detects a user operation acting on the album shortcut button, the target image can be displayed.

[0173] It is understandable that the shooting scenes in the above examples, in addition to shooting of group photos of people, can also be scenes of shooting multiple animals, multiple plants or multiple target objects obtained by arrangement and combination, which will not be elaborated here.

[0174] It should be noted that the above user interfaces are merely some examples provided for this application and should not be regarded as limitations of this application.

[0175] The following describes the specific implementation of the above embodiment in conjunction with a group photo scene. Figure 8The image processing method shown may include:

[0176] S800: The camera application receives a request to start the camera application.

[0177] In this embodiment, the request to start the camera application can be used to request to start the camera application. Optionally, the electronic device can send a request to start the camera application to the camera application when detecting an operation that triggers the camera application to start. Figure 3 As shown, when the electronic device detects a user operation on the camera application icon 101 of the user interface 100, a request for starting the camera application is sent to the camera application.

[0178] It can be understood that the above examples are examples of some camera applications receiving requests to start the camera application, and the present embodiment is not limited thereto.

[0179] S802: The camera application sends a request to the main camera to start the main camera.

[0180] S804: The main camera starts and collects second preview image data.

[0181] The second preview image data may refer to preview image data collected after the camera application is started.

[0182] S806: The main camera sends second preview image data to the camera HAL.

[0183] S808: The camera HAL processes the second preview image data and sends the processed image to the display screen for display.

[0184] Since the second preview image data is the image data collected by the main camera when the camera application is started, the second preview image is displayed when the camera application is started. Figure 4A As shown, the second preview image may be a preview image displayed in the preview window 201 of the user interface 200 .

[0185] It is understandable that the above provides examples of some of the second preview images, and the second preview image actually displayed is related to the orientation of the main camera when it is activated, and this embodiment is not limited to this.

[0186] S810: The camera application receives a request to start the group photo mode.

[0187] The request to start the group photo mode may be used to request to enter the group photo mode. Optionally, the electronic device may send a request to start the group photo mode to the camera application when detecting a trigger operation to enter the group photo mode. For example, Figure 4AAs shown, when the electronic device detects a user operation acting on the group photo mode option 2021, a request to turn on the group photo mode is sent to the camera application. Figure 4B and Figure 4C As shown, when the electronic device detects a user operation on more buttons, it may enter Figure 4C In the user interface 300 shown, when the electronic device detects a user operation on the group photo mode option 2021, it sends a request to the camera application to start the group photo mode.

[0188] It is understandable that the above examples are examples of some camera applications receiving requests to start the group photo mode, and the present embodiment is not limited thereto.

[0189] S812: The camera application sends a request to the telephoto camera to start the telephoto camera.

[0190] In this embodiment, the telephoto camera has a longer focal length than the main camera, so the corresponding field of view angle is lower, but the corresponding image clarity is also higher.

[0191] S814: The telephoto camera is started and first preview image data is collected.

[0192] The first preview data may be understood as preview image data collected after entering the group photo mode.

[0193] S816: The telephoto camera sends first preview image data to the camera HAL.

[0194] S818. The camera HAL processes the first preview image data and sends the processed image to a display screen for display.

[0195] In this embodiment, since the first preview image data is image data collected by the telephoto camera when the telephoto camera is activated, when entering the group photo mode, the first preview image is displayed. Figure 5A As shown, the first preview image may be a preview image displayed in the preview window 201 of the user interface 400 .

[0196] It is understandable that the above provides examples of some of the first preview images, and the first preview image actually displayed is related to the orientation of the telephoto camera when it is activated, and this embodiment is not limited to this.

[0197] In another possible implementation, when entering the group photo mode, the first preview image data may also be collected by the main camera. Figure 5B As shown, the first preview image may also be a preview image displayed in the preview window 201 of the user interface 400 .

[0198] It is understandable that by starting the telephoto camera when entering the group photo mode, the telephoto camera can be called in time to capture multiple first images, thereby improving the efficiency of capturing multiple first images and further improving the efficiency of image enhancement.

[0199] S820: The camera application receives a click operation of a capture button.

[0200] The click operation of this embodiment may be one or more times, which is not limited here. In other possible implementations, the click operation may also be a user operation such as a touch operation or a voice control operation.

[0201] S822: The camera application sends a request to the telephoto camera to capture a first image.

[0202] S824. The telephoto camera captures a first image.

[0203] Wherein, each time the telephoto camera receives a request to shoot a first image, it shoots a first image.

[0204] Exemplarily, the telephoto camera is movable, such as Fig. 6A and Figure 6B As shown, after receiving a click operation of the shooting button, the telephoto camera is controlled to move, and a request to shoot the first image is sent to the telephoto camera at a set interval. Each time the telephoto camera receives a request to shoot the first image, it shoots a first image. Since the telephoto camera is in the process of moving, when the telephoto camera receives a request to shoot the first image at different time points, the position of the telephoto camera is not fixed. In this way, the telephoto camera can shoot multiple first images at different viewing angles, that is, the image contents of the multiple first images are different. In this way, the multiple first images can capture all the characters. Optionally, if the first image is shot by the telephoto camera, the first image can also be a telephoto image.

[0205] Optionally, the telephoto camera may move in a manner including, but not limited to, translation or rotation around a fixed axis. Exemplarily, the telephoto camera may move in a first direction and / or a second direction, the first direction and the second direction being perpendicular, so that the electronic device may capture multiple first images in different postures, for example, capturing multiple first images in a horizontal or vertical posture. Exemplarily, the telephoto camera may rotate around a first fixed axis and / or around a second fixed axis, the first fixed axis and the second fixed axis being perpendicular, so that the electronic device may also capture multiple first images in different postures.

[0206] Another example is Figure 6C and Fig.6DAs shown, each time a click operation of the shooting button is received, a request to shoot the first image is sent. In this way, the user can move the electronic device each time after shooting a first image, so that each time the telephoto camera receives a request to shoot the first image, its position is different, so that the telephoto camera can shoot multiple first images at different viewing angles.

[0207] Optionally, the multiple first images of this embodiment may be as follows: Fig. 6A and Figure 6B The preview window 201 shown displays a low-viewing-angle and high-definition image.

[0208] S826: The telephoto camera sends the multiple first images to the camera application.

[0209] In this embodiment, the camera application may save a plurality of first images so as to perform image enhancement based on the plurality of first images.

[0210] S828: The camera application receives a click operation of the Done button.

[0211] In this embodiment, when the camera application receives a click operation of the completion button, it means that the user believes that the multiple first images can include all the persons, and the shooting of the first image can be ended at this time.

[0212] S830: The camera application sends a request to the main camera to capture a second image.

[0213] In this embodiment, the camera application may send a request to the main camera to capture the second image when receiving a click operation of the completion button; or when receiving a click operation of the completion button, the camera application may send a request to the main camera to capture the second image. Figure 7 The illustrated example shows a request to the main camera to capture a second image when the user operates the capture button.

[0214] S832: The main camera captures a second image.

[0215] For example, the second image may be Figure 7 Since the second image has a lower definition than the first image, the first image may also be a low-resolution image (LR image).

[0216] S834: The main camera sends the second image to the camera application.

[0217] In another possible implementation, the second image may be captured by an ultra-wide-angle camera.

[0218] S836: The camera application stitches the multiple first images to obtain a stitched image, and uses the stitched image to enhance the person in the second image to obtain a target image.

[0219] In this embodiment, the main purpose of enhancing the person is to make the person clearer, that is, the clarity of the person in the target image is higher than the clarity of the person in the second image.

[0220] It should be noted that in some other possible implementations, the second image may be captured by the main camera first, and then multiple first images may be captured by the telephoto camera. In this way, when entering the group photo mode from the normal photo mode, the preview image data can continue to be collected by the main camera, thereby reducing the frequent switching of the camera, and then reducing the frequent switching of the preview image on the preview interface, which can improve the user experience. In addition, it can also reduce the time wasted due to switching the camera back and forth, thereby improving the efficiency of image processing.

[0221] It should be noted that, in some other possible implementations, the first image may be captured by a combination of one or more cameras, as long as the first image has a higher definition and a lower field of view than the second image, this embodiment does not limit how the first image is captured. Similarly, the second image may be captured by a combination of one or more cameras, as long as the second image has a larger field of view and a lower definition than the first image, this embodiment does not limit how the second image is captured.

[0222] Based on the above embodiments, Fig. 9 The specific implementation of how to perform image enhancement is explained.

[0223] 1. The first images are stitched to obtain a stitched image.

[0224] In this embodiment, a plurality of first images may be stitched together to obtain a stitched image including a complete group portrait area. It is understandable that if the first image is a telephoto image, the stitched image may also be understood as a telephoto stitched image.

[0225] Optionally, the image stitching method of this embodiment can be image stitching based on key point matching. For example, it can include but is not limited to the following steps of image preprocessing: including basic operations of digital image processing (such as denoising, edge extraction, histogram processing, etc.), establishing a matching template for the image, and performing some transformation (such as Fourier transform, wavelet transform, etc.) on the image. Then, image registration is performed: a certain matching strategy is adopted to find the corresponding position of the template or feature point in the image to be stitched in the reference image, and then determine the transformation relationship between the two images. Then, a transformation model is established: according to the corresponding relationship between the template or image features, the parameter values ​​in the mathematical model are calculated, so as to establish a mathematical transformation model of the two images. Then, a unified coordinate transformation is performed: according to the established mathematical transformation model, the image to be stitched is transformed into the coordinate system of the reference image to complete the unified coordinate transformation. Then, fusion reconstruction is performed: the overlapping areas of the images to be stitched are fused to obtain a smooth and seamless panoramic image of stitching and reconstruction.

[0226] 2. Perform portrait segmentation on the second image to obtain a portrait mask.

[0227] The portrait mask may be a mask corresponding to the entire portrait area, that is, the portrait mask is a mask of the entire portrait area. In addition, the portrait mask may refer to a mask corresponding to each portrait, which is not limited here.

[0228] Optionally, the portrait segmentation method of this embodiment can be performed through a specific portrait segmentation algorithm. The portrait segmentation algorithm mainly includes two categories: semantic segmentation and instance segmentation. Semantic segmentation refers to assigning each pixel in the image to a specific category, such as human body, background, etc.; while instance segmentation is to group pixels belonging to the same entity together on the basis of semantic segmentation to form independent instances. Optionally, the second image can be segmented by a trained portrait segmentation model. Specifically, it can be a portrait segmentation model based on deep learning training, and any neural network with sufficient accuracy can be used.

[0229] It should be noted that the portrait area of ​​this embodiment may include the face area and the body area. If the portrait area includes the face area and the body area, the object area to be super-resolved is also the face area and the body area. In this way, the problem of obvious difference in clarity between the face and the body after enhancement can be avoided.

[0230] 3. Rectangularize the image.

[0231] The portrait mask and the stitched image are input into the image rectangularization module to rectangularize the reference image. That is, the stitched image can be rectangularized based on the portrait mask through the image rectangularization module to obtain the reference image.

[0232] In this embodiment, the purpose of performing image matrixing is mainly to perform correction processing, such as removing black edges around the spliced ​​image, reshaping the spliced ​​image into a rectangular size, etc. Optionally, after the portrait area in the spliced ​​image is reshaped, the size of the object area in the reference image matches the size of the object area mask.

[0233] Optionally, the splicing is put together with the portrait mask for feature extraction. The obtained feature map is passed through a grid regression network and a grid motion and bending (warp, also known as distortion) process to obtain a first-stage output representing a coarse-scale network result. The first-stage result is input into the grid regression network again for grid motion prediction. The obtained result grid is moved and warped to obtain the final rectangular result.

[0234] Among them, features can be extracted by a feature extractor. Optionally, convolutional pooling blocks are stacked to extract high-level semantic features from the input. Exemplarily, 8 convolutional layers are used, and the number of filters is set to 64, 64, 64, 64, 128, 128, 128, and 128 respectively. A maximum pooling layer is used after the 2nd, 4th, and 6th convolutional layers.

[0235] Among them, the network regression processing can be performed through a mesh motion regressor. Optionally, after feature extraction, an adaptive pooling layer is used to fix the resolution of the feature map. Subsequently, a fully convolutional structure is used as a mesh motion regressor to predict the horizontal and vertical motion of each vertex based on a regular mesh. Assuming the mesh resolution is U×V, the size of the output volume is (U+1)×(V+1)×2.

[0236] Among them, the grid motion can be estimated based on the residual progressive regression. Optionally, the accurate grid motion can be estimated in a progressive way. First, the warped image is not directly used as the input of the new network, because this will double the computational complexity. Instead, we warp the intermediate feature map. Then, two regressors with the same structure are used to predict the primary grid motion and the residual grid motion respectively. Then, the feature map can also be warped.

[0237] In this embodiment, the spliced ​​image can be rectangularized based on the portrait mask, that is, the portrait mask is used as a reference benchmark. In this way, the rectangularization effect is better and the efficiency is higher.

[0238] In another possible implementation, the mesh regression network, mesh movement, and wrapping process may be performed once, or at least three times. The process may be set as needed and is not limited here.

[0239] 4. Reference-based super-resolution (RefSR).

[0240] RefSR may mean that, in addition to the low-resolution image, there is also a high-resolution reference image, which has similar texture or content to the image to be restored. Using the information of the Ref image can help the SR process.

[0241] In this embodiment, semantic guidance of portrait information can be added, super-resolution and texture migration can be separated, a separate branch can be used to extract high-frequency texture information related to the portrait, and the second image can be super-resolved. Then, the high-frequency texture information is migrated to the super-resolved second image, and finally the portrait super-resolution result (target image) can be obtained.

[0242] Exemplarily, the second image can be processed using a general SR network to obtain a single-frame super-resolution result. Then, feature extraction is performed on the LR image and the reference image respectively to obtain reference image features of the reference image and second image features of the second image. Then, feature alignment is performed guided by the portrait mask information to obtain a corresponding pixel-level offset map. The pixel-level offset map can indicate the offset information of each pixel in the second image in the reference image, such as indicating the offset direction and offset amount of each pixel in the second image in the reference image.

[0243] Then, the reference image is encoded to obtain a reference feature map (also called encoded feature). Then, the pixel-level offset map is applied to the variable convolution operation or attention-based convolution operation on the encoded feature map to obtain the texture features to be migrated. Then, the obtained texture features to be migrated and the single-frame super-resolution result (the second image after the single-frame super-resolution) are input into the texture migration module to obtain the final portrait super-resolution result.

[0244] In this embodiment, the offset direction and amount of each pixel of the second image in the reference image can be determined based on the pixel-level offset map. In this way, the variable convolution operation or the attention-based convolution operation on the encoded feature map takes into account the pixel offset between the reference image and the second image. In this way, the correlation between the obtained migrated texture features and the object area is higher.

[0245] Optionally, the aforementioned image super-resolution process can be performed based on a pre-trained super-resolution model. In this embodiment, the pre-trained super-resolution model may include a feature extraction network for extracting features from a reference image and a second image respectively, a feature alignment network for aligning features of the reference image and features of the second image based on an object region mask, an encoder for encoding the reference image, a convolution network for performing a deformable convolution operation or an attention-based convolution operation on a reference feature map based on a pixel-level offset map, a texture migration network for migrating texture features to be migrated to the second image, and a single-frame super-resolution network for performing single-frame super-resolution on the second image.

[0246] Optionally, the super-resolution model to be trained can be trained based on the training samples until the training of the super-resolution model to be trained is completed to obtain a pre-trained super-resolution model. In this embodiment, the super-resolution model to be trained may also include a decoder, and the decoder is used to decode texture features to obtain texture information, and the texture information can be used to supervise the training process of the super-resolution model to be trained. Specifically, the training samples may include training image samples, and the baseline texture information can be determined based on the training image samples. Then, the training loss can be calculated based on the decoded texture information and the baseline texture information, and then it is determined whether the training of the super-resolution model to be trained is completed. If the training is not completed, the model parameters of the super-resolution model to be trained are updated and the training is continued.

[0247] It should be noted that the super-resolution model to be trained may include modules or networks related to texture information extraction, while other modules unrelated to texture information extraction may directly select trained modules or networks. Optionally, in this embodiment, the modules related to texture information extraction may include but are not limited to at least one of a feature extraction network, a feature alignment network, an encoder, and a convolutional network.

[0248] In another possible implementation, the shape of the reference image may be processed into a shape other than a rectangle, such as a circle, etc., as long as it can correct the portrait area in the stitched image, and there is no limitation on the specific shape of the reference image.

[0249] In another possible implementation, the spliced ​​image may not be rectangularized, that is, the spliced ​​image is used as a reference image to perform the super-resolution process.

[0250] It can be understood that by rectangularizing the stitched image, the reference image is more closely matched to the portrait mask, and the image is rectangularized based on the stitched image and the portrait area mask, giving priority to minimizing the loss of the portrait effect after rectangularization, so that the result of the super-resolution processing is more accurate and efficient. In addition, if the reference image is a stitched image generated based on a telephoto image, and then super-resolution is performed based on the stitched telephoto image, the portrait area texture of the telephoto image can be applied to the super-resolution process to improve the authenticity of the super-resolution result.

[0251] In another possible implementation, in addition to super-resolution of the portrait area, super-resolution may be performed on the object area of ​​interest other than the portrait area, for example, super-resolution may be performed on the landscape. In addition, if the image captured is a non-portrait image, such as an animal image or a plant image, super-resolution may also be performed on the object area of ​​interest such as the animal or plant, which is not limited here.

[0252] It should be noted that image enhancement can refer to purposefully emphasizing the overall or local characteristics of an image, making an originally unclear image clear or emphasizing certain interesting features, expanding the differences between the features of different objects in the image, suppressing uninteresting features, so as to improve image quality, enrich information, enhance image interpretation and recognition effects, and meet the needs of certain special analysis. In this embodiment, in addition to super-resolution, certain interesting features can also be emphasized to enhance the prominent content of the image and improve the image display effect.

[0253] The following specific embodiments are used to describe the technical solutions of the present application in detail. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0254] The image processing method provided in this embodiment may include:

[0255] The first interface is displayed, and the first interface includes a first preview image and a first button.

[0256] In response to a first operation on the first button, prompt information and a second button are displayed, wherein the prompt information is used to prompt the acquisition of images of multiple target objects.

[0257] In the target time period, a plurality of first images are captured by adopting the first mode, and the plurality of first images are displayed respectively.

[0258] In this embodiment, how to trigger the first mode to capture the first image can be referred to the relevant description of FIG. 5 , which will not be elaborated here.

[0259] In response to a second operation on a second button, a second image is captured using a second mode, wherein the second button is continuously displayed within a target time period, the target time period includes a time period from receiving the first operation to receiving the second operation, and a clarity of the image captured using the first mode is higher than a clarity of the image captured using the second mode, and a field of view of the image captured using the first mode is lower than a field of view of the image captured using the second mode.

[0260] In this embodiment, how the first mode captures the first image and how the second mode captures the second image can be referred to the relevant description of FIG. 5 , which will not be elaborated herein.

[0261] The target object in the second image is enhanced by using a stitched image to obtain a target image, wherein the stitched image includes an image obtained by stitching a plurality of first images.

[0262] In this embodiment, how to stitch multiple first images to obtain a stitched image and how to enhance the stitched image to obtain a target image can refer to the relevant description of FIG. 5 , and is not limited here.

[0263] In another possible implementation, the image processing method further includes:

[0264] Displaying a second interface, the second interface includes a second preview image, a third button and a fourth button, the third button is in an unselected state, the fourth button is in a selected state, and the second preview image is a preview image captured in the mode of the fourth button;

[0265] Accordingly, the first interface is displayed, including:

[0266] In response to the third operation on the third button, a first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

[0267] In another possible implementation, the image processing method further includes:

[0268] Displaying a third interface, the third interface includes a second preview image, a fourth button and a fifth button, the fourth button is in a selected state, the second preview image is a preview image captured in the mode of the fourth button, and the fifth button is in an unselected state;

[0269] In response to a fourth operation on the fifth button, a fourth interface is displayed, where the fourth interface includes a third button, and the third button is in an unselected state;

[0270] The first interface is displayed, including:

[0271] In response to the third operation on the third button, a first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

[0272] In another possible implementation, the first interface may be directly displayed when the camera application is opened, that is, the default mode when the camera application is opened is the group photo mode, which can be set as needed and is not limited here.

[0273] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0274] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0275] The image processing method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above list sorting method.

[0276] The image processing method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above related descriptions, which will not be repeated here.

[0277] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0278] like Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip 1000 includes one or more (including two) processors 1001 , a communication line 1002 , a communication interface 1003 and a memory 1004 .

[0279] In some implementations, the memory 1004 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0280] The method described in the above embodiment of the present application can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 1001. The above processor 1001 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1001 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0281] The steps of the method described in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1004, and the processor 1001 reads the information in the memory 1004 and completes the steps of the above method in combination with its hardware.

[0282] The processor 1001 , the memory 1004 , and the communication interface 1003 may communicate with each other via the communication line 1002 .

[0283] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0284] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0285] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0286] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0287] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0288] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Displaying a first interface, wherein the first interface includes a first preview image and a first button; In response to a first operation on the first button, displaying prompt information and a second button, wherein the prompt information is used to prompt the user to capture images of multiple target objects; Within a target time period, a plurality of first images are obtained by shooting in a first mode, and the plurality of first images are displayed respectively; In response to a second operation on the second button, a second image is captured using a second mode, wherein the second button is continuously displayed within the target time period, the target time period includes a time period from when the first operation is received to when the second operation is received, a definition of the image captured using the first mode is higher than a definition of the image captured using the second mode, and a field of view angle of the image captured using the first mode is lower than a field of view angle of the image captured using the second mode; A target object in the second image is enhanced by using a stitched image to obtain a target image, wherein the stitched image includes an image obtained by stitching the multiple first images.

2. The method according to claim 1, characterized in that The prompt information includes information for prompting the user to keep the posture of the electronic device unchanged, and the plurality of first images are obtained by the electronic device taking pictures through a movable camera; Alternatively, the prompt information includes information for guiding a moving trajectory, and the plurality of first images are captured by moving the electronic device.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Displaying a second interface, the second interface including a second preview image, a third button and a fourth button, the third button is in an unselected state, the fourth button is in a selected state, and the second preview image is a preview image captured in the mode of the fourth button; The displaying of the first interface includes: In response to a third operation on the third button, the first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Displaying a third interface, the third interface including a second preview image, a fourth button and a fifth button, the fourth button is in a selected state, the second preview image is a preview image captured in the mode of the fourth button, and the fifth button is in an unselected state; In response to a fourth operation on the fifth button, displaying a fourth interface, the fourth interface including a third button, the third button being in an unselected state; The displaying of the first interface includes: In response to a third operation on the third button, the first interface is displayed, in which the third button is in a selected state and the fourth button is in an unselected state.

5. The method according to any one of claims 1 to 4, characterized in that The step of enhancing the target object in the second image by using the stitched image to obtain the target image includes: Segmenting the second image to obtain an object region mask; Correcting the stitched image based on the object region mask to obtain a reference image, wherein a size of the object region in the reference image matches a size of the object region mask; The second image is super-resolved based on the reference image and the object region mask to obtain a target image.

6. The method according to claim 5, characterized in that The super-resolution of the second image based on the reference image and the object region mask to obtain a target image includes: Extracting features from the reference image and the second image respectively to obtain reference image features of the reference image and second image features of the second image; Aligning the reference image features with the second image features based on the object region mask to obtain a pixel-level offset map, wherein the pixel-level offset map indicates offset information of each pixel in the second image in the reference image; Encoding the reference image to obtain a reference feature map; Performing a deformable convolution operation or an attention-based convolution operation on the reference feature map based on the pixel-level offset map to obtain a texture feature to be migrated; The texture features to be migrated are migrated to the second image to obtain a target image.

7. The method according to claim 6, characterized in that The method further comprises: Performing single-frame super-resolution on the second image to obtain a second image after single-frame super-resolution; The step of migrating the texture features to be migrated to the second image to obtain a target image includes: The texture features to be migrated are migrated to the second image after single-frame super-resolution to obtain a target image.

8. The method according to claim 6 or 7, characterized in that: The target image is obtained by super-resolving the reference image, the object region mask and the second image through a pre-trained super-resolving model, wherein the pre-trained super-resolving model includes a feature extraction network, a feature alignment network, an encoder, a convolutional network and a texture migration network; The feature extraction network is used to extract features from the reference image and the second image respectively to obtain reference image features of the reference image and second image features of the second image; The feature alignment network is used to perform feature alignment on the reference image features and the second image features based on the object region mask to obtain a pixel-level offset map; The encoder is used to encode the reference image to obtain a reference feature map; The convolutional network is used to perform a deformable convolution operation or an attention-based convolution operation on the reference feature map based on the pixel-level offset map to obtain a texture feature to be migrated; The texture migration network is used to migrate the texture features to be migrated to the second image to obtain a target image.

9. The method according to claim 8, characterized in that The target image is obtained by migrating the texture features to be migrated to a second image after single-frame super-resolution, and the pre-trained super-resolution model also includes a single-frame super-resolution network; The single-frame super-resolution network is used to perform single-frame super-resolution on the second image to obtain a second image after single-frame super-resolution; The texture migration network is used to migrate the texture features to be migrated to the second image after single-frame super-resolution to obtain a target image.

10. The method according to any one of claims 5 to 9, characterized in that: The step of correcting the spliced ​​image based on the object area mask to obtain a reference image includes: Splicing the spliced ​​image with the object area mask, and performing feature extraction on the spliced ​​image with the object area mask to obtain a spliced ​​feature map; Performing N correction processes based on the spliced ​​feature map to obtain a reference image, where N is an integer not less than 2; Each calibration process includes: Perform network regression on the input, as well as mesh motion and bending processing to obtain the output; Among them, the input of the first correction process is the stitched image with the object area mask stitched, the input of the Mth correction process is the output obtained by the M-1th correction process, the output obtained by the Nth correction process is used as the reference image, 1<M≤N, and M is an integer.

11. An image processing method, characterized in that: include: Acquire a second image and a plurality of first images, wherein the plurality of first images are obtained by shooting in the first mode, and the second image is obtained by shooting in the second mode, wherein the clarity of the image shot in the first mode is higher than the clarity of the image shot in the second mode, and the field of view of the image shot in the first mode is lower than the field of view of the image shot in the second mode; splicing the plurality of first images to obtain a spliced ​​image; The target object in the second image is enhanced by using the stitched image to obtain a target image.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 11.

13. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 11.

Images