Image processing method and electronic device

Abstract

The present application provides an image processing method and an electronic device, wherein the electronic device includes a main camera and a telephoto camera, wherein the main camera and / or the telephoto camera are rotatable cameras. When shooting with the main camera and the telephoto camera, the clarity of the fused image can be improved. The method may include: opening a camera application; receiving a first operation on a shooting control in the camera application; determining a main camera cropped image and a telephoto cropped image in response to the first operation; determining an enhanced feature image based on the main camera cropped image, the telephoto cropped image and an image processing model; generating a fused image based on the enhanced feature image, and displaying the fused image. Among them, the main camera cropped image is obtained by cropping the main camera image shot by the main camera, and the telephoto cropped image is obtained by cropping the telephoto image shot by the telephoto camera.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] The camera module of an electronic device may include one or more cameras, such as one or more of a main camera, a wide-angle camera, a telephoto camera, etc. The main camera is the most important camera in the camera module, with the highest pixel, the largest aperture, and the best imaging quality, and can be used to shoot most shooting scenes. The shooting range or field of view of the wide-angle camera is wider than that of the main camera, and can be used to shoot wide scenes such as landscapes and buildings. The shooting distance of the telephoto camera is farther than that of the main camera, and can be used to shoot distant scenery, people, etc. The wide-angle camera and the telephoto camera can make up for the shortcomings of the main camera, thereby improving the shooting effect. For example, the main camera and the telephoto camera are fused: by using the main camera and the telephoto camera to shoot the same portrait in the distance, a main camera portrait image and a telephoto portrait image can be obtained (although the main camera has high pixels, the effect of shooting portraits in the distance is not good, and the telephoto camera shoots the distance through lossless zoom, which can obtain a clearer portrait, so the clarity of the telephoto portrait image is better than that of the main camera portrait image). The telephoto portrait image is fused into the main camera portrait image to obtain a fused portrait image, and the overall clarity and portrait clarity of the fused portrait image are higher than those of the main camera portrait image. The fusion of the main camera and the telephoto camera is a multi-camera fusion method.

[0003] At present, each camera in the camera module is a fixed camera, that is, the physical position of each camera on the electronic device is fixed, and the position angle difference between the cameras is a fixed value. For the fusion of the main camera and the telephoto camera, there may be a difference between the shooting angle of the main camera and the shooting angle of the telephoto camera, which will affect the clarity of the fused image. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can improve the clarity of a fused image.

[0005] In a first aspect, the present application provides an image processing method, which can be applied to an electronic device including a main camera and a telephoto camera, wherein the main camera and / or the telephoto camera are rotatable cameras. The method may include: opening a camera application and displaying a user interface of the camera application, the user interface including shooting controls; receiving a first operation on the shooting controls of the camera application, that is, receiving a first operation on the shooting controls in the user interface; in response to the first operation, performing the following operations: determining a main camera cropped image and a telephoto cropped image; determining an enhanced feature image based on the main camera cropped image, the telephoto cropped image and an image processing model; generating a fused image based on the enhanced feature image and displaying the fused image.

[0006] By implementing the image processing method provided in the first aspect, in a scenario where the main camera and / or the telephoto camera is a rotatable camera, an enhanced feature image with higher clarity can be obtained by using an image processing model, and a fused image can be generated based on the enhanced feature image, thereby improving the clarity of the fused image.

[0007] Among them, the rotatable camera refers to a camera that can be rotated up, down, left, and right at a certain angle, similar to a rotatable surveillance camera. Optionally, the rotatable camera can not only be rotated up, down, left, and right at a certain angle, but also can be moved forward and backward a certain distance. Since the rotatable camera can be rotated and / or moved, there is a difference in the shooting angle of the camera and other fixed cameras on the electronic device. For the main camera and / or the telephoto camera being a rotatable camera, there may be a large difference between the shooting angle of the main camera and the shooting angle of the telephoto camera, then there may be a large difference between the main camera portrait image taken by the main camera and the telephoto portrait image taken by the telephoto camera, such as at least one of the differences in position difference, angle difference, expression difference and color difference, and these differences will affect the clarity of the fused image. The method provided in the first aspect can reduce the impact of these differences on the clarity of the fused image, thereby improving the clarity of the fused image.

[0008] The image processing method provided in the first aspect can also be applied to the scene where both the main camera and the telephoto camera are fixed cameras. The image processing model can not only improve the clarity of the fused image, but also improve the efficiency of generating the fused image. For the case where both the main camera and the telephoto camera are fixed cameras, the fused image obtained by the method provided in the first aspect has higher clarity than the fused image obtained by the current multi-camera fusion.

[0009] The shooting range of a fixed camera is limited. If the user wants to shoot a scene near a shooting range through the camera module, the user needs to move the electronic device so that the camera module can reshoot the scene. However, if the camera is a rotatable camera, the user does not need to move the electronic device. By adjusting the camera, the camera can be rotated to a certain angle and / or moved forward and backward a certain distance to shoot the scene.

[0010] In combination with the method provided in the first aspect, in some embodiments, in response to the first operation, the main camera image is determined from the image captured by the main camera, and the telephoto image is determined from the image captured by the telephoto camera. That is, in response to the first operation, the main camera and the telephoto camera are called to capture images of the same shooting scene, the main camera image is determined from at least one image captured by the main camera for the shooting scene, and the telephoto image is determined from at least one image captured by the telephoto camera. After determining the main camera image and the telephoto image, the main camera image and the telephoto image are cropped to obtain a main camera cropped image and a telephoto cropped image, so that the main camera cropped image and the telephoto cropped image are input into the image processing model for processing. Compared with inputting the main camera image and the telephoto image into the image processing model for processing, inputting the main camera cropped image and the telephoto cropped image into the image processing model for processing can improve processing efficiency and focus on processing the cropped image to highlight the features of the cropped image.

[0011] In combination with the method provided in the first aspect, in some embodiments, when cropping the main camera image and the telephoto image, the region recognition model can be called to identify the preset regions in the main camera image and the telephoto image, and the cropping function can be called to crop the main camera image and the telephoto image based on the preset regions, respectively, to obtain the main camera cropped image and the telephoto cropped image. The preset region can be, for example, a portrait facial region, a portrait region, or a designated shooting region. The region recognition model can be, for example, a facial recognition model for identifying a facial region of a face. Since there are differences in the shooting parameters of the main camera image and the telephoto image, the size of the preset region in the main camera image is different from the size of the preset region in the telephoto image. The preset region is used to indicate a certain region, and the region size is not limited to be the same. The cropping function is used to crop the image and crop the preset region image from the image.

[0012] In combination with the method provided in the first aspect, in some embodiments, since the enhanced feature image is a cropped image, it is necessary to fuse the enhanced feature image into the main camera image to generate a fused image. That is, a fused image is generated based on the enhanced feature image and the main camera image. It is understandable that the main camera cropped image in the main camera image is replaced or overwritten with the enhanced feature image, and during the replacement or overwriting process, the surrounding image of the cropped position is subjected to peripheral fusion processing. In this way, the fused image has a higher clarity and more detailed details in the preset area than the main camera image.

[0013] In combination with the method provided in the first aspect, in some embodiments, an enhanced feature image is determined based on the main camera cropped image, the telephoto cropped image and the image processing model. The main camera cropped image and the telephoto cropped image are input into the image processing model to obtain the enhanced feature image output by the image processing model.

[0014] In combination with the method provided in the first aspect, in some embodiments, the image processing model is specifically used to perform alignment processing on the telephoto cropped image based on the main camera cropped image to obtain a first intermediate image, so that the first intermediate image is aligned with the main camera cropped image (for example, size alignment and pixel alignment) for subsequent processing; based on the main camera cropped image, the first intermediate image is color mapped to obtain a second intermediate image, so as to reduce the color difference between the second intermediate image and the main camera cropped image, thereby reducing the difference between the fused image and the main camera image; based on the second intermediate image, attention learning is performed on the main camera cropped image to obtain an enhanced feature image, that is, the main camera cropped image learns the second intermediate image to obtain an enhanced feature image, thereby improving the clarity of the enhanced feature image.

[0015] In combination with the method provided in the first aspect, in some embodiments, features of the first intermediate image are aligned with features of the main cropped image. That is, through the alignment process, the features of the first intermediate image and the features of the main cropped image can be aligned in size and pixels to facilitate color mapping and attention learning.

[0016] In combination with the method provided in the first aspect, in some embodiments, the color features of the main cropped image are fused into the second intermediate image. That is, the second intermediate image fuses the color features of the main cropped image to reduce the color difference between the second intermediate image and the main cropped image, thereby reducing the difference between the fused image and the main image.

[0017] In combination with the method provided in the first aspect, in some embodiments, before using the image processing model, the parameters of the image processing model are received, and based on the parameters of the image processing model, the image processing model is deployed so that the electronic device can call the image processing model to process the main camera cropped image and the telephoto cropped image. The parameters of the image processing model may come from a server or a personal computer (PC), etc., and the PC can be understood as a model training device. That is, the server or PC executes the model training process, and when a training model that meets the deployment conditions is obtained, the parameters of the training model are sent to the electronic device, so that the electronic device deploys the image processing model based on the parameters.

[0018] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0019] In a third aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0020] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] It can be understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided in the first aspect of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A-Figure 1D is a set of user interface schematic diagrams provided in an embodiment of the present application;

[0023] Figure 2A-2C It is an effect diagram of image processing provided by an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the software architecture provided by the embodiment of the present application;

[0025] Figure 4 is a flow chart of an electronic device provided in an embodiment of the present application performing image processing to obtain a fused image;

[0026] Figure 5 is an example diagram of cropping and alignment provided in an embodiment of the present application;

[0027] Figure 6 yes Figure 4 Schematic diagram of several implementations of step 404 and step 405;

[0028] Figure 7 is a schematic diagram of the training process of the image processing model provided in the embodiment of the present application;

[0029] Figure 8It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0031] 1. Alignment

[0032] For multiple cameras of an electronic device shooting the same scene, due to differences in the spatial position and sensor of the camera, the parameter information of different cameras is different, such as focal length, pixel size, field of view (FOV), etc., and thus the images obtained by different cameras shooting the same scene are different. For example, for a scene including a portrait, due to factors such as the pixel of the main camera is higher than that of the telephoto camera, the telephoto camera has a multiple lossless zoom, and the aperture of the telephoto camera is larger than the aperture of the main camera, the main camera image obtained by the main camera is compared with the telephoto image obtained by the telephoto camera. The main camera image shows more scenery and smaller portraits, and then the portraits in the main camera image and the telephoto image are magnified, and the clarity of the portraits in the telephoto image is higher than that in the main camera image. Among them, since the focal length of the main camera is shorter than that of the telephoto camera and the FOV of the main camera is larger than that of the telephoto camera, the telephoto image can be understood as a close-up image of the scene, and the main camera image can be understood as a distant image or a wide-angle image of the scene. Therefore, before these images are fused, these images are aligned. For example, taking the alignment processing of the telephoto image and the main camera image of the same scene as an example, for the same features in the same scene, the size of the telephoto image is larger than the size of the main camera image, and the clarity of the telephoto image is higher than the clarity of the main camera image. For example, for the facial features of the same portrait, the size of the facial features in the telephoto image is larger than the size in the main camera image, and the clarity in the telephoto image is higher than the clarity in the main camera image. Alignment may include size alignment and pixel alignment. Size alignment can reduce the size difference between different images; pixel alignment can reduce the pixel difference between different images.

[0033] 1) Dimension alignment

[0034] Size alignment can be understood as cropping and scaling other images based on the size of the reference image so that the processed other images are aligned with the reference image in size.

[0035] In one implementation, when the electronic device obtains parameter information of the telephoto image and the main camera image, it can calculate the scale difference between the telephoto image and the main camera image based on the parameter information, and then crop and enlarge the main camera image based on the scale difference, so that the processed main camera image can be aligned with the telephoto image in size.

[0036] In another way, the electronic device performs sparse feature point matching on the telephoto image and the main camera image without knowing the parameter information of the telephoto image and the main camera image to obtain the transformation matrix between them, and crops and enlarges the main camera image based on the transformation matrix so that the processed main camera image can be aligned with the telephoto image in size. For the convenience of description, the image obtained by size-aligning the main camera image is called a size-aligned image. Although the size-aligned image is aligned with the telephoto image in size, for the same object, the size-aligned image may be rotated and / or translated compared to the telephoto image, so the size-aligned image needs to be pixel-aligned.

[0037] 2) Pixel Alignment

[0038] Pixel alignment can be understood as rotating and / or translating other images based on the positions of each object in the reference image so that the processed other images are aligned with the reference image in terms of pixels. For the same object, the object in the other images moves or rotates relative to the object in the reference image.

[0039] Pixel alignment is usually implemented using an optical flow alignment algorithm, which can cover local operations and better handle detail changes, such as changes in facial expressions. In order to improve alignment accuracy and efficiency, a deep learning optical flow alignment algorithm can be used, such as the DISFlow algorithm in the open source computer vision library (Open Source Computer Vision Library, OpenCV).

[0040] In the above, the main camera image is aligned with the telephoto image as an example, that is, the telephoto image is used as a reference image, and the main camera image is adjusted to align with the telephoto image. The telephoto image can also be aligned with the main camera image, that is, the main camera image is used as a reference image, and the telephoto image is adjusted to align with the main camera image. For example, the telephoto image can be cropped and reduced so that the processed telephoto image can be aligned with the main camera image in size.

[0041] 2. Adaptive Instance Normalization (AdaIN)

[0042] The AdaIN algorithm is an image style transfer algorithm that can transfer the style and texture of one image (style image) to another image (content image) while retaining the main structure of the content image. The AdaIN algorithm is based on a feed-forward neural network model, with a fast generation speed. The AdaIN algorithm can support the transfer of any style, not just a specific style. You only need to input the content image and style image into the AdaIN algorithm to get a style transfer image.

[0043] Exemplarily, assuming that the feature of the content image is x and the feature of the style image is y, the AdaIN algorithm can transfer the style of y to x, as shown in the following formula.

[0044]

[0045] Among them, μ(y) represents the mean of the features of the style image, σ(y) represents the variance of the features of the style image, μ(x) represents the mean of the features of the content image, and σ(x) represents the variance of the features of the content image.

[0046] In an embodiment of the present application, the AdaIN algorithm is used to transfer color features in one image to another image to reduce color differences.

[0047] 3. Attention Mechanism

[0048] The attention mechanism originates from the study of human vision. In cognitive science, due to the bottleneck of information processing, humans selectively focus on part of all information while ignoring other visible information. The above mechanism is usually called the attention mechanism. Different parts of the human retina have different degrees of information processing capabilities, that is, acuity, and only the fovea of ​​the retina has the strongest acuity. In order to make rational use of limited visual information processing resources, humans need to select a specific part of the visual area and then focus on it. For example, when people read, usually only a small number of words to be read will be paid attention to and processed. In summary, the attention mechanism has two main aspects: determining the part that needs attention; allocating limited information processing resources to the part of attention.

[0049] The attention mechanism enables a neural network (model) to focus on a subset of its inputs (or features): select specific inputs. The attention mechanism can be applied to any type of input regardless of its shape. In the case of limited computing power, the attention mechanism is a resource allocation solution that is the main means to solve the problem of information overload, allocating computing resources to more important tasks.

[0050] In the embodiment of the present application, the attention mechanism is used to focus on learning the difference features of a specific area (such as an area of ​​interest) in the image to improve the ability to learn the details of the area of ​​interest. For example, the difference features of the facial features, hair and other areas in a portrait image are focused on learning.

[0051] 4. Loss parameter

[0052] The loss parameter refers to the difference between the predicted value (estimated value) and the actual value (expected value, reference value, label (ground truth)). "Loss" means the penalty imposed on the model for failing to produce the expected results. Its function is to determine the performance of the model by comparing the predicted output and the expected output of the model, and then find the optimization direction. If the deviation between the two is very large, the loss value will be large; if the deviation is small or the values ​​are almost the same, the loss value will be very low. Among them, the loss parameter can be the loss value of the loss function (Loss Function) or the loss value of the cost function (Cost Function).

[0053] For a certain model, if its loss parameters meet the corresponding conditions, then the model can be deployed and / or used. In the embodiment of the present application, during the training process of the image processing model, if its loss parameters meet the corresponding deployment conditions, then the image processing model under the loss parameters can be deployed in an electronic device, and the electronic device calls the image processing model to process the main camera image and the telephoto image.

[0054] First, the application scenarios of the embodiments of the present application are introduced. The image processing method and electronic device provided by the embodiments of the present application can be applied to the scenario where the main camera is a fixed camera and the telephoto camera is a rotatable camera, or the scenario where the main camera is a rotatable camera and the telephoto camera is a fixed camera, or the scenario where both the main camera and the telephoto camera are rotatable cameras, or the scenario where both the main camera and the telephoto camera are fixed cameras. Among them, a fixed camera refers to a camera whose physical position on the electronic device is fixed, and the position angle difference between the fixed cameras is a constant value, and a rotatable camera refers to a camera that can be rotated up and down, left and right by a certain angle and / or moved forward and backward by a certain distance, so that the position angle difference between it and the fixed camera is within a range of variation.

[0055] For cameras where the main camera and / or the telephoto camera are rotatable, taking portraits as an example, since there may be a large difference between the main camera portrait image taken by the main camera and the telephoto camera portrait image taken by the telephoto camera, the use of the current multi-camera fusion algorithm will affect the display effect of the fused image, such as affecting the clarity. Furthermore, the use of the embodiments of the present application can improve the clarity of the fused image. For cameras where both the main camera and the telephoto camera are fixed, the use of the embodiments of the present application can further improve the clarity of the fused image compared to the use of the current multi-camera fusion algorithm.

[0056] The image processing method provided in the embodiment of the present application can be applied to an electronic device with the image processing capability, such as a mobile phone, a tablet computer, etc.

[0057] By implementing the image processing method provided in the embodiment of the present application, the electronic device 100 can perform alignment processing, color mapping processing and attention learning to obtain a fused image. Performing color mapping processing helps to reduce the color difference between the fused image and the main camera image. Performing attention learning helps to improve the clarity of the fused image. Thus, even if the telephoto camera is not fixed, the color difference can be reduced and the clarity of the fused image can be improved. Taking the shooting of a scene including a portrait as an example, the implementation of the embodiment of the present application can improve the clarity of the portrait in the fused image.

[0058] Exemplarily, the electronic device 100 may include a telephoto camera and a main camera. In some embodiments, the electronic device 100 calls the telephoto camera and the main camera to capture images of the same scene, obtains a telephoto image and a main camera image, and processes the main camera image based on the telephoto image to improve the clarity of the fused image. In other embodiments, the electronic device 1 calls the telephoto camera and the main camera to capture a video of the same scene, obtains an original telephoto video frame and an original main camera video frame, and processes the original main camera video frame based on the original telephoto video frame to improve the clarity of the original fused video frame. This process is repeated to obtain a fused video stream with higher clarity.

[0059] The electronic device 100 is not limited to mobile phones and tablet computers. It can also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0060] Figure 1A-Figure 1D An exemplary set of user interfaces displayed on the electronic device 100 is shown below. Figure 1A-Figure 1D The application scenarios of the image processing method provided in the embodiments of the present application are specifically introduced.

[0061] first, Figure 1A The user interface of the electronic device 100 displaying the installed applications, i.e., the home page, is shown as an example. Figure 1A As shown, one or more application icons are displayed in the main page, such as a "clock" application icon, a "calendar" application icon, a "weather" application icon, and the like.

[0062] The one or more application icons include an icon of a "camera" application, namely, icon 111. The electronic device 100 can detect a user operation on icon 111, such as a click operation. In response to the user operation, the electronic device 100 can turn on the camera, call the camera to capture an image, and display the image. Figure 1B The user interface shown.

[0063] In another implementation, the lock screen user interface of the electronic device 100 displays a "camera" icon. The electronic device 100 can detect a user operation on the "camera" icon, such as clicking and pulling up to the top of the lock screen user interface. In response to the user operation, the electronic device 100 can turn on the camera, call the camera to capture an image, and display the image. Figure 1B The user interface shown.

[0064] When the electronic device 100 turns on the camera, one camera may be turned on by default, such as the telephoto camera or the front camera, or multiple cameras may be turned on by default, such as the main camera and the telephoto camera. Optionally, when the focal length of the turned-on main camera is within a certain range, the electronic device 100 also turns on the telephoto camera.

[0065] Figure 1B and Figure 1C The user interface when the “camera” application is running on the electronic device 100 is exemplarily shown. The user interface may include a window 121, a shooting control 122, an image display control 123, and a reversal control 124.

[0066] Among them, the window 121 is used to display the image collected by the camera, which can be understood as the display area of ​​the collected image, so that the user can input a click operation to the shooting control 122 or the reversal control 124 according to the picture displayed in the window 121. In some embodiments, the window 121 also includes some controls, such as a focus adjustment control. When the electronic device 100 detects a user operation acting on the focus adjustment control, it can adjust the focus of the camera in response to the user operation, for example, from 1x to 2x, and the type of the camera is not limited here.

[0067] When one camera is turned on by default in the electronic device 100, the image displayed in the window 121 is the image captured by the camera. When two cameras are turned on in the electronic device 100, the image displayed in the window 121 may be the image captured by the main camera, or may be the image fused with the image captured by the main camera and the image captured by the telephoto camera, depending on the processing capability of the electronic device 100, and the former requires lower processing capability.

[0068] When the electronic device 100 detects a user operation acting on the shooting control 122, in response to the user operation, the captured image can be determined from the captured image, and a thumbnail of the captured image can be displayed in the image display control 123. For example, the main camera image is determined from the image captured by the main camera, and the telephoto image is determined from the image captured by the telephoto camera. The main camera image and the telephoto image have the same frame time. The main camera image is processed based on the telephoto image to obtain a fused image. The thumbnail of the fused image can be displayed in the image display control 123. Then, when the user operation acting on the image display control 123 is detected, in response to the user operation, the main camera image can be displayed in the image display control 123. Figure 1D The user interface shown in the figure displays the fused image; the fused image can also be saved in a gallery, and then when a click operation acting on the fused image in the gallery is detected, in response to the click operation, Figure 1DThe user interface shown displays a fused image.

[0069] When the electronic device 100 detects a user operation on the inversion control 124, in response to the user operation, it can turn off the rear camera and turn on the front camera, and call the front camera to capture images; or, it can turn off the front camera and turn on the rear camera, and call the rear camera to capture images.

[0070] like Figure 1B and Figure 1C The user interface shown also includes a "night scene" mode, a "portrait" mode, a "photo" mode, and a "video recording" mode. These modes are used for example and do not constitute a limitation on the embodiments of the present application. For example, in actual applications, a "multi-lens video recording" mode is also included. Among them, the "video recording" mode is used to record video files, and the "multi-lens video recording" mode is used to record video files when the front camera and the rear camera are turned on at the same time. The "night scene" mode is used to capture night scene images to improve the clarity of night scene images. The "portrait" mode is mainly used to capture portrait images. The image processing method provided in the embodiments of the present application can be used in this mode. The "photo" mode is used to capture single-frame images. Figure 1B and Figure 1C In the video, the mode selected by the user is the "Portrait" mode.

[0071] When the electronic device 100 detects a user operation acting on the image display control 123, in response to the user operation, the electronic device 100 displays the following Figure 1D The user interface is a user interface for browsing pictures, and may include an image display window 131 for displaying an image or video. The image may be a fused image obtained by using the image processing method provided in the embodiment of the present application, and the video may be a fused video obtained by using the image processing method provided in the embodiment of the present application.

[0072] The user interface may also include a control 134 , a share control 135 , a favorite control 136 , an edit control 133 , a delete control 137 , and the like.

[0073] The control 134 may be used to display detailed information of the image, such as shooting time, shooting location, color encoding format, bit rate, frame rate, pixel size, etc.

[0074] The sharing control 135 can be used to send the displayed image or video to other applications for use. For example, after detecting a user operation on the sharing control, in response to the user operation, the electronic device 100 can display one or more application icons, and the one or more application icons include the icon of social software 1. After detecting a user operation on the application icon of social software 1, in response to the user operation, the electronic device 100 can send the image or video to the social software 1, and further, the user can share the above image or video with friends through the social software.

[0075] The favorites control 136 can be used to mark an image or video. Figure 1D In the user interface shown, when a user operation acting on the favorites control is detected, in response to the user operation, the electronic device 100 may mark the displayed image or video as a user's favorite image or video. The electronic device 100 may generate an album for displaying the images or videos marked as the user's favorite.

[0076] The delete control 137 may be used to delete the displayed image or video.

[0077] By using the image processing method provided in the embodiment of the present application, the image display control 123 can display a thumbnail of the fused image, or save the fused image in a gallery. Figure 1D The image displayed in the image display window 131 is a fused image. The fused image has higher definition than the main image, for example, the definition of the facial area is particularly obvious, and the color difference between the fused image and the main image is small.

[0078] For example, see Figure 2A-2C The image shown, Figure 2A The image shown is a main camera image captured by the main camera (i.e., a main camera image determined from an image captured by the main camera). Figure 2B The image shown is a telephoto image captured by a telephoto camera (i.e., a telephoto image determined from an image captured by the telephoto camera). Figure 2C The image shown is a fused image obtained by using the image processing method provided in the embodiment of the present application. Figure 1D The image display window 131 in the display Figure 2C The fused image shown.

[0079] Since the pixels of the main camera are higher than those of the telephoto camera, the telephoto camera has a lossless zoom of multiples, and the aperture of the telephoto camera is larger than that of the main camera, when the main camera and the telephoto camera shoot the same distant scene, the main camera image shows more scenery and smaller portraits than the telephoto image, and thus the clarity of the portrait in the main camera image is lower than that in the telephoto image. Figure 2B The face resolution is much higher than Figure 2A Because the focal length of the main camera is shorter than that of the telephoto camera and the FOV of the main camera is larger than that of the telephoto camera, Figure 2B It can be understood as a close-up image of the scene. Figure 2A It can be understood as a distant view of the scene. Figure 2A and Figure 2C It can be seen that Figure 2C The clarity is higher than Figure 2A clarity. Figure 2C The clarity is not as good as Figure 2B of clarity, but higher than Figure 2A clarity.

[0080] Figure 2A-2C In the figure, the facial expression of the person does not change. However, because the telephoto camera and / or the main camera are rotatable cameras, the shooting angles of the telephoto camera and the main camera may be different. For example, there is a 10° angle deviation between the shooting angles. Therefore, in fact Figure 2A Facial expressions and Figure 2B There may be differences in facial expressions in the images. By using the embodiment of the present application, even if there are differences, they can be processed, thereby improving the clarity of the fused image.

[0081] The following describes in detail the electronic device 100 Figure 1D The specific process of the user interface is shown.

[0082] first, Figure 3 The software architecture of the electronic device 100 is exemplarily shown.

[0083] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0084] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer (HAL), and the kernel layer.

[0085] The application layer can include a series of application packages. Figure 3As shown, the application package may include camera, gallery, video, music, navigation, calendar, map, WLAN and other applications. In the embodiment of the present application, the camera can provide the following during operation: Figures 1B to 1D In the user interface shown, the electronic device 100 can display a thumbnail of the fused image in the image display control 123, and can also display the fused image in the image display window 131. After starting the camera, the electronic device 100 can call the camera to capture an image. The electronic device 100 includes at least two cameras, for example, one camera is a main camera, and the other camera is a telephoto camera, and the telephoto camera is rotatable.

[0086] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer includes a camera service framework, and the camera service framework may include an image processing model. Optionally, the image processing model processes the main camera cropped image and the telephoto cropped image to obtain an enhanced feature image. Optionally, the image processing model processes the main camera image and the telephoto image to obtain a fused image. The image processing model can realize the functions of alignment processing, color mapping processing and attention learning.

[0087] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0088] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

[0089] The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0090] The hardware abstraction layer (HAL) is an interface layer between the kernel layer and the hardware, which can be used to control the actions of the hardware. The hardware abstraction layer may include hardware modules for controlling the actions of the hardware.

[0091] The kernel layer is the foundation of the Android system. For example, ART relies on the kernel layer to perform underlying functions, such as threads and low-level memory management. The kernel layer is a layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, a GPU driver, etc. In an embodiment of the present application, the kernel layer includes a camera driver and a display driver. The camera driver is used to drive the camera to capture images, and the display driver is used to drive the display screen to display fused images.

[0092] based on Figure 3 The software architecture shown in the figure illustrates the image processing method provided in the embodiment of the present application. In some embodiments, the electronic device 100 detects a user operation acting on the camera application, responds to the user operation, and starts the camera so that the camera captures images. Upon detecting the user operation, the camera application triggers an instruction to start the camera. The camera application calls the API interface of the application framework layer to send the instruction to the camera service framework, and the camera service framework calls the hardware abstraction layer to send the instruction to the camera driver. The camera driver can drive the camera to start and drive the camera to capture images. The images captured by the camera can be cached in the image buffer area.

[0093] When the electronic device 100 starts the camera, one camera or multiple cameras may be started. In the embodiment of the present application, starting the main camera and the telephoto camera is taken as an example, and the main camera and / or the telephoto camera may be a rotatable camera. Optionally, the electronic device 100 detects the user operation and immediately starts the main camera and the telephoto camera. Optionally, the electronic device 100 detects the user operation and the main camera is already started, and the telephoto camera may be further started. Optionally, the electronic device 100 detects the user operation and the focal length of the main camera is within a certain range, and the telephoto camera may be further started.

[0094] The main camera can transmit the main image to the image processing model in the camera service framework, and the telephoto camera can transmit the telephoto image to the image processing model in the camera service framework. Optionally, the image processing model processes the main image and the telephoto image to obtain a fused image. Optionally, the camera service framework crops the main image and the telephoto image respectively to obtain a main cropped image and a telephoto cropped image, and the image processing model processes the main cropped image and the telephoto cropped image to obtain an enhanced feature image. The image processing model or the camera service framework calls a post-processing algorithm to process the enhanced feature image to generate a fused image. The camera service framework calls the hardware abstraction layer to transmit the fused image to the display screen so that the display screen displays the fused image. The fused image has higher definition than the main image.

[0095] Figure 4 The flowchart of the electronic device 100 performing image processing to obtain a fused image is shown as an example. Figure 1A-Figure 1D The user interface shown and Figure 3 The software architecture of the electronic device 100 is shown. Below, the embodiment of the present application will specifically introduce the process of the electronic device 100 performing image processing to obtain a fused image.

[0096] 401 , the electronic device 100 determines a main image and a telephoto image in response to a first operation on a shooting control.

[0097] The main camera image and the telephoto image are images of the same scene, and the shooting parameters of the two are different, such as different focal lengths. In the embodiment of the present application, the main camera image and the telephoto image are images of a target portrait as an example, where the target portrait is any portrait, and the main camera image can be, for example, Figure 2A The image shown, the telephoto image can be seen for example Figure 2B Because the pixels of the main camera are higher than those of the telephoto camera, the telephoto camera has a lossless zoom, and the aperture of the telephoto camera is larger than that of the main camera, the main camera image shows more scenery and smaller portraits than the telephoto image, so the clarity of the portrait in the main camera image is lower than that in the telephoto image. Figure 2B The face resolution is much higher than Figure 2A Because the focal length of the main camera is shorter than that of the telephoto camera and the FOV of the main camera is larger than that of the telephoto camera, Figure 2B It can be understood as a close-up image. Figure 2A It can be understood as a long-range image. Figure 2A and Figure 2B Taking the target portrait as an example with a white background, in actual applications, the image of the target portrait has a non-white background. The embodiment of the present application mainly processes the target portrait area, and a similar method can also be used to process the background part.

[0098] The shooting control is a control in the camera application. When receiving a click operation, the electronic device 100 can respond to the operation by determining a preview image from at least one frame of image captured by the camera, and displaying a thumbnail of the preview image in the image display control of the camera application. The electronic device 100 starts the camera application and detects whether a first operation on the shooting control is received, such as receiving a first operation on the shooting control. Figure 1B A click operation on the shooting control 122 in the user interface shown. Upon receiving the click operation on the shooting control, the electronic device 100 determines a main image and a telephoto image.

[0099] Optionally, in response to the click operation, the electronic device 100 calls the main camera and the telephoto camera to shoot the same shooting scene to obtain a main camera image and a telephoto image.

[0100] Optionally, in response to the click operation, the electronic device 100 determines a main camera image from at least one frame of image captured by the main camera for a certain shooting scene, and determines a telephoto image from at least one frame of image captured by the telephoto camera for the shooting scene. When the electronic device 100 receives a user operation on a camera application, it starts the camera application, and starting the camera application can trigger the start of the camera, thereby calling the camera to capture images. Before detecting a click operation on the shooting control, the camera has been capturing images and caching the captured images in an image cache. When a click operation on the shooting control is detected, in response to the click operation, the main camera image is determined from at least one image captured by the main camera cached in the image cache, and the telephoto image is determined from at least one image captured by the telephoto camera cached in the image cache. Starting the camera application can be done by Figure 1A The user operation trigger shown may also be triggered by a user operation on other applications, for example, by a user operation on a shooting icon of an instant messaging application.

[0101] When the electronic device 100 detects a user operation on the shooting control, the camera application triggers a shooting instruction. The camera application calls the API interface of the application framework layer to send the shooting instruction to the camera service framework, and the camera service framework calls the camera interface to read the cached image from the image buffer area, and determines the main camera image and the telephoto image from the cached image.

[0102] At 402 , the electronic device 100 crops the main camera image and the telephoto image to obtain a main camera cropped image and a telephoto cropped image.

[0103] Since the photographing effect of the photographing object in the telephoto image is better than that of the photographing effect of the photographing object in the main camera image, the photographing object in the main camera image can be processed with reference to the photographing object in the telephoto image to improve its photographing effect. Then, the electronic device 100 crops the main camera image and the telephoto image, crops out the photographing object, and obtains the main camera cropped image and the telephoto cropped image. For example, the photographing effect of the portrait in the telephoto image is better than that of the portrait in the main camera image, so the portrait can be cropped from the main camera image and the portrait can be cropped from the telephoto image. Among them, the photographing object is an example, and can also be replaced by a partial photographing area, and the partial photographing area can be, for example, a designated photographing area. The electronic device 100 can identify the photographing object or the partial photographing area from the main camera image and the telephoto image, and crop based on the recognition result to obtain the main camera cropped image and the telephoto cropped image. Among them, the area where the photographing object is located or the partial photographing area can be considered as a preset area.

[0104] Exemplarily, the electronic device 100 crops the main camera image based on the facial area of ​​the target portrait to obtain a main camera cropped image, and crops the telephoto image based on the facial area of ​​the target portrait to obtain a telephoto cropped image. In this way, both the main camera cropped image and the telephoto cropped image are images of the facial area of ​​the target portrait. The sizes of the main camera cropped image and the telephoto cropped image may be different, for example, the size of the main camera cropped image is smaller than the size of the telephoto cropped image.

[0105] During the implementation process, the camera service framework can call the region recognition model to identify the preset regions in the main camera image and the telephoto image. For example, the facial recognition model can be called to identify the portrait facial region in the main camera image and the telephoto image, and the cropping function can be called to crop the main camera image and the telephoto image based on the portrait facial region to obtain the main camera cropped image and the telephoto cropped image. The region recognition model can be included in the image processing model or can be independent of the image processing model. The cropping function can be included in the image processing model or can be independent of the image processing model.

[0106] For example, see Figure 5 As shown in the example diagram of cropping and alignment, the electronic device 100 identifies the portrait facial area in the main camera image and the telephoto image, and crops the main camera image and the telephoto image based on the portrait facial area to obtain a main camera cropped image and a telephoto cropped image.

[0107] 403 , the electronic device 100 performs alignment processing on the telephoto cropped image based on the main camera cropped image to obtain a first intermediate image.

[0108] The electronic device 100 performs alignment processing on the telephoto cropped image based on the main camera cropped image to obtain a first intermediate image, which can be understood as transforming (warping) the telephoto cropped image so that the first intermediate image obtained by warping is aligned with the main camera cropped image. By warping the telephoto cropped image, the features in the main camera cropped image can be learned, so that the facial area features of the first intermediate image compared with the main camera image do not change, for example, the features such as facial expressions and facial features positions in the first intermediate image are aligned with the features in the main camera image. Warp refers to transforming an image into the perspective of another image, or moving an image to a target position according to relative motion to align with another image. Warp can also be understood as alignment processing, and the alignment processing may include size alignment processing and pixel alignment. Size alignment is to adjust the size of the telephoto cropped image so that the size of the first intermediate image is the same as that of the main camera cropped image. Pixel alignment is to rotate and / or translate each facial feature in the telephoto cropped image based on the position of each facial feature in the main camera cropped image, so that the first intermediate image is aligned with the main camera cropped image in terms of pixels.

[0109] Aligning the telephoto cropped image based on the main camera cropped image can cover local motion and better handle the expression difference between the main camera cropped image and the telephoto cropped image.

[0110] For example, see Figure 5 In the example diagram of cropping and alignment shown in FIG. 1 , the electronic device 100 performs an alignment process on the telephoto cropped image based on the main camera cropped image to obtain a first intermediate image. The first intermediate image is aligned with the main camera cropped image in terms of size and pixels.

[0111] 404 , the electronic device 100 performs color mapping processing on the first intermediate image based on the main cropped image to obtain a second intermediate image.

[0112] The color mapping process can be implemented by the AdaIN algorithm. The color mapping process of the first intermediate image based on the main cropped image can be understood as migrating the color features of the main cropped image to the first intermediate image. Exemplarily, assuming that the color feature of the first intermediate image is i and the color feature of the main cropped image is j, the AdaIN algorithm can migrate j to i, as shown in the following formula.

[0113]

[0114] Among them, μ(j) represents the mean of the color features of the main cropped image, σ(j) represents the variance of the color features of the main cropped image, μ(i) represents the mean of the color features of the first intermediate image, σ(i) represents the variance of the features of the first intermediate image, and AdaIN(i,j) represents the color features of the second intermediate image.

[0115] There will be color differences between images taken by different cameras. In order to avoid the color of the fused image being changed, color mapping is used to reduce the color difference between the second intermediate image and the main camera cropped image, thereby reducing the color difference between the fused image and the main camera image.

[0116] 405 , the electronic device 100 performs attention learning on the main cropped image based on the second intermediate image to obtain an enhanced feature image.

[0117] Since the main camera and / or the telephoto camera are rotatable cameras, there may be a large difference between the shooting angle of the main camera and the shooting angle of the telephoto camera. For example, the angles for shooting faces may be different, and thus the facial expressions and the positions of the facial features in the main image and the telephoto image may be different. Therefore, it is necessary to find the corresponding facial partitions to learn the detailed information of the facial features or hair areas in a more targeted manner. In order to avoid the mismatch of the features of different regions of the main image and the telephoto image, attention learning is added. For example, image A is a facial image of a target portrait taken at a straight angle by the main camera, and image B is a facial image of a target portrait taken by a telephoto camera rotated 10°. Then the target portraits in images A and B will differ in position, angle, expression, etc. Attention learning can learn the feature maps with large differences between images A and B, so as to enhance the areas with large differences in image A, so as to improve the clarity of image A and reflect more details while ensuring that the features of image A remain unchanged.

[0118] The electronic device 100 performs attention learning on the main camera cropped image based on the second intermediate image, that is, the electronic device 100 learns the features with large differences between the second intermediate image and the main camera cropped image, and enhances these features in the main camera cropped image to obtain an enhanced feature image. In other words, the main camera cropped image can obtain an enhanced feature image by learning the second intermediate image. Since the second intermediate image is an image obtained by aligning and color mapping the telephoto cropped image, the clarity of the second intermediate image is higher than that of the main camera cropped image and the color difference between the second intermediate image and the main camera cropped image is small, and the second intermediate image and the main camera cropped image have differences in angle and expression, so by learning the second intermediate image through the main camera cropped image, the clarity of the enhanced feature image can be improved and more details can be reflected.

[0119] In one implementation, the electronic device 100 may input the second intermediate image and the main cropped image into an attention learning module to obtain an enhanced feature image output by the attention learning module. The attention learning module may implement the functions of an attention learning model, or may be described as an attention learning module including an attention learning model. The attention learning model may focus on learning specific areas (such as facial features and hair) where there are differences between the second intermediate image and the main cropped image, so as to enhance the calculation weight of the features of the specific area, help improve the clarity of the specific area, and reflect more details of the specific area.

[0120] Step 404 and step 405 may be implemented in any of the following ways:

[0121] Mode 1: After the first intermediate image is input into the encoder, the encoder may output the features of the first intermediate image; after the main camera cropped image is input into the encoder, the encoder may output the features of the main camera cropped image. The features of the first intermediate image and the features of the main camera cropped image are input into the color mapping module, and the color mapping module may output the second intermediate image, and the features of the second intermediate image incorporate the color features of the main camera cropped image. The features of the second intermediate image and the features of the main camera cropped image are input into the attention learning module, and the attention learning module may output the enhanced features of the main camera cropped image. The enhanced features of the main camera cropped image are input into the decoder, and the decoder may output the enhanced feature image.

[0122] Mode 2: After the first intermediate image is input into the encoder, the encoder may output the features of the first intermediate image; after the main camera cropped image is input into the encoder, the encoder may output the features of the main camera cropped image. The features of the first intermediate image and the features of the main camera cropped image are input into the color mapping module, and the color mapping module may output the second intermediate image, and the features of the second intermediate image incorporate the color features of the main camera cropped image. The features of the second intermediate image are input into the decoder, and the decoder may output the second intermediate image. The second intermediate image and the main camera cropped image are input into the attention learning module, and the attention learning module may output the enhanced feature image.

[0123] Mode 3: After the first intermediate image is input into the encoder, the encoder may output the features of the first intermediate image; after the main cropped image is input into the encoder, the encoder may output the features of the main cropped image. The features of the first intermediate image and the features of the main cropped image are input into the color mapping module, and the color mapping module may output the second intermediate image. The features of the second intermediate image incorporate the color features of the main cropped image. The features of the second intermediate image are input into the decoder, and the decoder may output the second intermediate image. The second intermediate image is input into the encoder, and the encoder may output the features of the second intermediate image. The features of the second intermediate image and the features of the main cropped image are input into the attention learning module, and the attention learning module may output the enhanced features of the second intermediate image. The enhanced features of the second intermediate image are input into the decoder, and the decoder may output the enhanced feature image.

[0124] Methods 1 to 3 introduce an encoder to obtain the features of the image, and introduce a decoder to obtain the image based on the features. The difference between Method 1 and Method 2 is the position of the decoder. In Method 1, the decoder is after the attention learning module, so that the color mapping process and attention learning are based on the features of the image, which is convenient to implement and has high processing efficiency; in Method 2, the decoder is before the attention learning module, so that the attention learning is based on the image, which has high processing complexity and low processing efficiency. Method 3 adds a pair of encoders and decoders on the basis of Method 1, which has a more complex structure, higher cost, and slow processing efficiency.

[0125] In methods 1 to 3, the first intermediate image and the main cropped image are respectively input into the encoder, which can be input into the same decoder or different encoders. The encoder can use a trained Visual Geometry Group (VGG) model. Correspondingly, the decoder can also use the VGG model. In method 3, the encoder can be the same encoder or different encoders.

[0126] For example, methods 1 to 3 can be found in Figure 6 As shown, Figure 6 Take the example of inputting the first intermediate image and the main cropped image into different encoders respectively.

[0127] Steps 403 to 405 can be understood as the electronic device 100 determining the enhanced feature image according to the main camera cropped image, the telephoto cropped image and the image processing model. That is, the electronic device 100 inputs the main camera cropped image and the telephoto cropped image into the image processing model, and can obtain the enhanced feature image output by the image processing model. Among them, the image processing model is used to implement alignment processing, color mapping processing and attention learning. That is, the image processing model is used to align the main camera cropped image based on the telephoto cropped image to obtain a first intermediate image; perform color mapping processing on the first intermediate image based on the main camera cropped image to obtain a second intermediate image; perform attention learning on the main camera cropped image based on the second intermediate image to obtain an enhanced feature image. The image processing model can be understood as a network model, which can implement the functions of alignment processing, color mapping processing and attention learning. The image processing model processes the main camera cropped image with the telephoto cropped image as a reference to obtain an enhanced feature image corresponding to the main camera cropped image.

[0128] During the implementation process, the camera service framework deploys an image processing model. The camera service framework inputs the main camera cropped image and the telephoto cropped image into the image processing model. The image processing model processes the main camera cropped image with reference to the telephoto cropped image, so that the image processing model can output an enhanced feature image.

[0129] 406 , the electronic device 100 generates a fused image based on the enhanced feature image.

[0130] Since the enhanced feature image is the enhanced feature image corresponding to the main camera cropped image, the main camera cropped image is a partial area image of the main camera image, and other area images in the main camera image except the main camera cropped image are not processed by the electronic device 100, the enhanced feature image is fused into the main camera image to obtain a fused image. Fusion of the enhanced feature image into the main camera image can be understood as posting the enhanced feature image back to the main camera image, and performing peripheral fusion processing during the posting process to better integrate the enhanced feature image into the main camera image. Exemplarily, according to the cropping position of the main camera cropped image in the main camera image, the enhanced feature image is posted back to the main camera image, and the surrounding images of the cropping position can be subjected to peripheral fusion processing during the posting process. Posting the enhanced feature image back to the main camera image can be understood as replacing or covering the main camera cropped image in the main camera image with the enhanced feature image. Peripheral fusion processing is a processing method involved in the image stitching process, which can reduce the difference in the stitching area. Generating a fused image can also be understood as generating a preview image, which is a fused image obtained based on the main camera image and the telephoto image.

[0131] During the implementation, the camera service framework may call a post-processing algorithm to fuse the enhanced feature image into the main camera image to obtain a fused image. The post-processing algorithm may include but is not limited to a reply algorithm and a peripheral fusion processing algorithm.

[0132] 407 , the electronic device 100 displays the fused image.

[0133] Optionally, the electronic device 100 displays the fused image in the image display area. The image display area may be, for example, Figure 1B or Figure 1C The display area shown in the image display control 123 may also be Figure 1D The display area shown in the image display window 131 in FIG. The fused image has more obvious details and a higher definition of the facial area than the main image.

[0134] During the implementation process, the camera service framework calls the hardware abstraction layer to transmit the fused image to the display driver. The display driver receives the fused image and drives the display screen to display the fused image.

[0135] exist Figure 4 In the illustrated embodiment, the color difference between the fused image and the main image can be reduced through color mapping processing; and the clarity and details of the fused image can be improved through attention learning processing.

[0136] Figure 4 In the illustrated embodiment, the electronic device 100 has the capabilities of alignment processing, color mapping processing, and attention learning. An image processing model is deployed in the electronic device 100, and the model is used to provide alignment processing, color mapping processing, and attention learning capabilities, that is, for executing steps 403 to 405. The camera service framework of the electronic device 100 can deploy the image processing model. The image processing model deployed by the electronic device 100 can come from a server. That is, during the process of training the model, the server can obtain an image processing model that meets the deployment conditions, and send the parameters of the model to the electronic device 100, so that the electronic device 100 deploys the image processing model based on these parameters, so that the electronic device 100 can call the image processing model to process the main camera cropped image based on the telephoto cropped image to obtain an enhanced feature image.

[0137] The following is an explanation of the training process of the image processing model. Figure 7 Flowchart shown. Figure 7 The flowchart shown may include:

[0138] 1, crop training image 1 into training cropped image 1, and crop training image 2 into training cropped image 2.

[0139] Among them, training image 1 and training image 2 are training images for the same shooting scene in the training image set, for example, both are training images of the target portrait. Training image 1 can be a training image of a telephoto image, and training image 2 is a training image of a main camera image. It can be understood that training image 1 and training image 2 have the same image identifier (identifier, ID), and the image ID can be used to identify images with different shooting parameters of the same shooting scene. Images with the same image ID have the same shooting scene and the same shooting object, but different shooting parameters, such as shooting angle, focal length, color, etc.

[0140] The training image set can be obtained in any of the following ways:

[0141] Method A: perform data screening on a public data set (such as VGG face 2), screen out training image sets with the same image ID, and distinguish the training image sets to obtain images with at least one different shooting angle, focal length, color, etc.

[0142] Method B: construct a training image set based on a public dataset. For example, if there is only one image in the public dataset for a certain image ID, then an image with at least one different shooting angle, focal length, color, etc. can be constructed based on the image to generate a training image set for the image. Optionally, a rendering (e.g., Rotate-and-Render) method can be used to construct the training image set.

[0143] Method C, data degradation. For a certain image, use a network learning method (such as a DSN network learning method, a full-image or patch training method, etc.), and superimpose at least one method of up and down sampling, JPEG compression, Gaussian blur, sinc filter, random small-scale distortion, etc. to degrade the image and obtain at least one degraded image. The image can be used as a telephoto image, and the degraded image can be used as the main image. The image can be an actually shot image. Data degradation is to simulate the degradation effect of an actually shot image.

[0144] Methods A to C can be executed by a server or manually, that is, a research and development personnel selects a training image set and inputs it into the server.

[0145] 2. Input the training cropped image 1 and the training cropped image 2 into the alignment module, align the training cropped image 1 based on the training cropped image 2, and the alignment module outputs the first training image.

[0146] The alignment processing of the training cropped image 1 based on the training cropped image 2 may be to transform the training cropped image 1 so that the first training image obtained by the transformation is aligned with the training cropped image 2. The alignment may include size alignment and pixel alignment. For size alignment, please refer to the specific description of "1) size alignment" above, and for pixel alignment, please refer to the specific description of "2) pixel alignment" above, which will not be repeated here.

[0147] 3. Input the first training image into the encoder, and the encoder can output the features (including color features) of the first training image; input the training cropped image 2 into the encoder, and the encoder can output the features (including color features) of the training cropped image 2.

[0148] The encoder for inputting the first training image and the encoder for inputting the training cropped image 2 may be the same encoder or different encoders. The encoder may adopt a VGG model.

[0149] 4. Input the features of the first training image and the features of the training cropped image 2 into the color mapping module for color mapping processing, and the color mapping module can output the features of the second training image. The features of the second training image incorporate the color features of the training cropped image 2.

[0150] The color mapping module may use the AdaIN algorithm to perform color mapping processing, and fuse the color features of the training cropped image 2 into the features of the second training image, so as to reduce the color difference between the second training image and the training cropped image 2.

[0151] Exemplarily, assuming that the color feature of the first training image is m, and the color feature of the training cropped image 2 is n, the AdaIN algorithm can transfer n to m, as shown in the following formula.

[0152]

[0153] Among them, μ(n) represents the mean of the color features of training cropped image 2, σ(n) represents the mean of the color features of training cropped image 2, μ(m) represents the mean of the color features of the first training image, σ(m) represents the mean of the features of the first training image, and AdaIN(m,n) represents the color features of the second training image.

[0154] 5. Input the features of the training cropped image 2 and the features of the second training image into the attention learning module to perform attention learning. The attention learning module can output the enhanced features of the training cropped image 2.

[0155] That is, the attention learning module learns the features with large differences between the second training image and the training cropped image 2, and enhances these features in the training cropped image 2 to obtain enhanced features.

[0156] 6. Input the enhanced features of the training cropped image 2 into the decoder, and the decoder can output the enhanced training image.

[0157] The decoder may adopt a VGG model. The decoder is used to decode the features to obtain an image, that is, to decode the enhanced features of the training cropped image 2 to obtain an enhanced training image.

[0158] 7. A fused training image can be generated based on the enhanced training image. That is, according to the cropping position of the training cropped image 2 in the training image 2, the enhanced training image is pasted back to the training image 2, and in the pasting process, the surrounding images of the cropping position can be subjected to peripheral fusion processing.

[0159] Since the enhanced training image is the enhanced image corresponding to the training cropped image 2, the training cropped image 2 is a partial area image of the training image 2, and other area images in the training image 2 except the training cropped image 2 have not been processed, the enhanced training image is fused into the training image 2 to obtain a fused training image.

[0160] Steps 2 to 7 may be performed by the server. Steps 2-7 may be a training process of the training model, that is, after the training cropped image 1 and the training cropped image 2 are input into the training model, a fused training image output by the training model may be obtained. Alternatively, steps 1-7 may be a training process of the training model, that is, after the training image 1 and the training image 2 are input into the training model, a fused training image output by the training model may be obtained. When the training model outputs the fused training image, it also outputs at least one of the following loss functions: layer 1 (Layer 1, L1) loss based on the whole image, loss containing key points prior in a specific area (such as a face) (i.e., landmark loss), perceptual loss, Generative Adversarial Network (GAN) loss capable of generating detailed textures, color loss (i.e., color loss), random up and down sampling loss (i.e., random up loss and / or random down loss), artifacts map loss to avoid GAN loss from generating too many defects, multi-scale loss of each layer feature output, etc. Among them, the L1 loss based on the whole image refers to the loss function between the fused training image and the target image. The target image is the image corresponding to the training image 2, which is used to measure whether the training model meets the requirements. The multi-scale loss of each layer of feature output refers to the feature loss of each layer of convolution output by the encoder after the fused training image and the target image are input into the same encoder. The multi-scale loss of each layer of feature output is used to supervise the training model in the process of generating intermediate features (such as the features of the first training image, the features of the second training image, and the enhanced features of the training cropped image 2).

[0161] When the above loss functions all meet the deployment conditions, the server may send the parameters of the training model to the electronic device 100 so that the electronic device 100 deploys the image processing model based on the parameters. Optionally, after the electronic device 100 deploys the image processing model, the main camera cropped image and the telephoto cropped image are input into the image processing model, and the image processing model may output an enhanced feature image. Optionally, after the electronic device 100 deploys the image processing model, the main camera image and the telephoto image are input into the image processing model, and the image processing model may output a fused image.

[0162] In the above, the server performs model training and sends the parameters of the training model that meets the deployment conditions to the electronic device as an example. In addition to the server, the PC can also perform model training and send the parameters of the training model that meets the deployment conditions to the electronic device. Other devices can also perform model training, which is not limited in the embodiments of the present application.

[0163] Optionally, the parameters of the image processing model can also be preset on the electronic device. For example, the parameters of the image processing model can be preset when the electronic device leaves the factory, so that when the electronic device is turned on or the camera application is used for the first time, the image processing model can be deployed based on the parameters of the image processing model; or, when the electronic device receives a trigger operation to deploy the image processing model, the image processing model can be deployed based on the parameters of the image processing model; and so on.

[0164] After deploying the image processing model, the electronic device may receive update parameters or patches from the server, where the update parameters are used to update the image processing model, and the patches are used to repair or upgrade the image processing model. Optionally, after deploying the image processing model, the electronic device may repair the image processing model by itself. For example, when the system of the electronic device is updated, the electronic device may repair the image processing model by itself.

[0165] Figure 8 The schematic diagram of the hardware structure of the electronic device 100 is exemplarily shown.

[0166] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio processing module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0167] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0168] The processor 110 may include one or more processing units, for example, the processor 110 may include an AP, a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. In an embodiment of the present application, the processor 110 may deploy an image processing model and call the image processing model.

[0169] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0170] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0171] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0172] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0173] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0174] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0175] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0176] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0177] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0178] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor.

[0179] The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194. In some embodiments, the modem processor can be an independent device. In other embodiments, the modem processor can be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0180] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0181] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0182] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. In the embodiment of the present application, the display screen 194 is used to display a fused image.

[0183] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0184] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc. Non-volatile memory may include disk storage devices and flash memory.

[0185] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0186] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0187] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0188] The electronic device 100 can implement audio functions such as music playing and recording through the audio processing module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0189] The audio processing module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio processing module 170 can also be used to encode and decode audio signals. In some embodiments, the audio processing module 170 can be arranged in the processor 110, or some functional modules of the audio processing module 170 can be arranged in the processor 110.

[0190] Speaker 170A is also called "speaker". Electronic device 100 can listen to music or listen to hands-free calls through speaker 170A. Receiver 170B is also called "earpiece". When electronic device 100 receives a call or voice message, the voice can be received by placing receiver 170B close to the human ear. Microphone 170C is also called "microphone" or "microphone". When making a call or sending a voice message, the user can speak by placing the human mouth close to microphone 170C to input the sound signal into microphone 170C.

[0191] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0192] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. The air pressure sensor 180C is used to measure the air pressure. The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure the distance. The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the brightness of the ambient light. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect the temperature. The touch sensor 180K, also known as a "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect a touch operation on or near it. The bone conduction sensor 180M can obtain a vibration signal.

[0193] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0194] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0195] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0196] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (GTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, GTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0197] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0198] As used in the specification and appended claims of the present application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items. As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrase "when determining..." or "if (stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)", depending on the context.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0200] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: Applicable to an electronic device including a main camera and a telephoto camera, wherein the main camera and / or the telephoto camera is a rotatable camera; the method includes: Open the Camera app; receiving a first operation on a shooting control in the camera application; In response to the first operation, determining a main-photo cropped image and a telephoto cropped image; Determine an enhanced feature image based on the main cropped image, the telephoto cropped image and an image processing model; the image processing model is used to perform alignment processing on the telephoto cropped image based on the main cropped image to obtain a first intermediate image; perform color mapping processing on the first intermediate image based on the main cropped image to obtain a second intermediate image; perform attention learning on the main cropped image based on the second intermediate image to obtain the enhanced feature image; Based on the enhanced feature image, generating a fused image; The fused image is displayed.

2. The method according to claim 1, characterized in that The determining of the main camera cropped image and the telephoto cropped image includes: Determine a main camera image from the image captured by the main camera, and determine a telephoto image from the image captured by the telephoto camera; The main camera image and the telephoto image are cropped to obtain a main camera cropped image and a telephoto cropped image.

3. The method according to claim 2, characterized in that The step of cropping the main camera image and the telephoto image to obtain a main camera cropped image and a telephoto cropped image includes: The region recognition model is called to identify preset regions in the main camera image and the telephoto image, and a cropping function is called to crop the main camera image and the telephoto image based on the preset regions to obtain a main camera cropped image and a telephoto cropped image.

4. The method according to claim 2, characterized in that The step of generating a fused image based on the enhanced feature image comprises: A fused image is generated based on the enhanced feature image and the main image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining an enhanced feature image according to the main cropped image, the telephoto cropped image and an image processing model includes: The main-camera cropped image and the telephoto cropped image are input into the image processing model to obtain an enhanced feature image output by the image processing model.

6. The method according to claim 1, characterized in that Features of the first intermediate image are aligned with features of the main cropped image.

7. The method according to claim 1, characterized in that The color features of the main cropped image are fused into the second intermediate image.

8. The method according to claim 1, characterized in that The method further comprises: Parameters of the image processing model are received, and based on the parameters of the image processing model, the image processing model is deployed.

9. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method according to any one of claims 1 to 8 is executed.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-8.

11. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 8 is executed.

Images