Image registration method and related equipment

Abstract

An embodiment of the present application discloses an image registration method and related equipment, characterized in that the method includes: determining depth information of a first image, the depth information including M pixels on the first image, and a depth value of each of the M pixels; extracting N first depth images from the first image based on the depth information, wherein each of the N first depth images includes a corresponding first pixel set, the first pixel set includes multiple target pixels in the first image, and the difference in depth values ​​between the multiple target pixels is within a preset range; matching a second image for each of the first depth images from a target gallery to obtain N second images; and fusing the N second images to obtain a target image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method and related equipment. Background Art

[0002] In the field of mobile phone photography, due to the limitations of mobile phone imaging devices, in high-magnification scenarios, such as when shooting distant scenes, the images taken by mobile phones have problems such as low resolution and blurred details. At present, in order to improve the clarity of images, a high-definition reference frame that is highly consistent with the user's shooting content and shooting angle is used to restore the details of the mobile phone image by extracting high-frequency information on the high-definition reference frame, thereby improving the clarity of the mobile phone image.

[0003] In this process, the high-definition reference frame is determined based on the image registration technology (i.e., spatial matching of two images). In the prior art, the plane feature points in the low-definition image taken by the mobile phone are extracted to be registered with the high-definition image in the high-definition library, so as to find a high-definition image from the high-definition library that best matches the plane feature points of the low-definition image and use it as the high-definition reference frame, and then the details of the low-definition image can be restored based on the high-definition reference frame. However, in actual application, it is found that since the registration is based only on the plane feature points of the low-definition image, the visual differences between different objects are ignored. Therefore, it is difficult to directly match a high-definition image in the high-definition library that is highly consistent with the shooting angle and shooting content of the low-definition image randomly taken by the user. That is, in general, there will be a large pixel deviation between the high-definition image matched from the high-definition library and the low-definition image. Therefore, in the prior art, the details of the low-definition image can only be restored based on the high-definition image with a large pixel deviation, resulting in deformation of some areas of the image and other problems. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present application is to provide an image registration method and related equipment to match a low-definition image to a high-definition image with a smaller pixel deviation.

[0005] In a first aspect, an embodiment of the present application provides an image registration method, which is applied to an electronic device, and the method includes: determining depth information of a first image, the depth information including M pixels on the first image, and a depth value of each of the M pixels, where M is an integer greater than 1; extracting N first depth images from the first image based on the depth information, wherein each of the N first depth images includes a corresponding first pixel set, the first pixel set includes multiple target pixels in the first image, and the difference in depth values ​​between the multiple target pixels is within a preset range, and N is an integer greater than 1; matching a second image for each first depth image from a target gallery to obtain N second images; and fusing the N second images to obtain a target image.

[0006] In an embodiment of the present application, taking into account the visual deviation of image contents at different depth levels, the image contents at different depth levels can be extracted based on the depth information of the first image, and image registration is performed for the image contents at each depth level (i.e., a high-definition image is matched from the gallery for the image contents at each depth level according to the shooting content and the shooting angle) to obtain a high-definition image with the smallest pixel deviation from the image contents at each depth level. Finally, multiple high-definition images are fused to obtain a target image that matches the first image (i.e., the target image is used as the registration result of the first image), thereby reducing the pixel deviation between the registration result and the first image.

[0007] In some embodiments, the method further includes: acquiring the first image, and estimating the depth values ​​of the M pixels on the first image through a preset algorithm to obtain the depth information, or acquiring the depth information of the first image through a sensor.

[0008] In an embodiment of the present application, the depth value of each pixel point of the first image can be estimated through a preset algorithm to obtain the depth information of the first image, or when a device that shoots the first image has a depth sensor, the depth information of the first image can be directly obtained through the sensor, and image contents of different depth levels are extracted based on the depth information of the first image, and image registration is performed on the image content of each depth level to obtain a high-definition image with the smallest pixel deviation from the image content of each depth level. Finally, multiple high-definition images are fused to obtain a target image that matches the first image, thereby reducing the pixel deviation between the registration result and the first image.

[0009] In some embodiments, the electronic device includes a display screen and a camera module, and the method further includes: starting the camera module and displaying a first user interface on the display screen, wherein the first user interface displays a captured picture; detecting a first user operation, and in response to the first user operation, determining the captured picture currently in the first user interface as the first image.

[0010] In an embodiment of the present application, the image currently captured by the electronic device can be used as the first image (i.e., a low-definition image), and then a target image (which can also be understood as an image with the smallest pixel deviation) whose shooting content and shooting angle are as consistent as possible can be matched for the image. Furthermore, the details of the low-definition image can be restored based on the target image, thereby improving the clarity of the image captured by the electronic device.

[0011] In some embodiments, extracting N first depth images from the first image based on the depth information includes: based on the depth values ​​of each of the M pixels, dividing a plurality of target pixels whose depth value difference between any two pixels is within a preset range into a first pixel point set to obtain N first pixel point sets; and obtaining the N first depth images based on the N first pixel point sets.

[0012] In an embodiment of the present application, the first image may include image contents at N depth levels, and the image contents at one depth level may include multiple target pixels. Therefore, the pixels at the same depth level may be divided into a set as much as possible, and then N images at different depth levels may be obtained based on the N pixel sets. Furthermore, image registration may be performed on the image contents at each depth level (i.e., a high-definition image is matched from the gallery for the image contents at each depth level according to the shooting contents and shooting angles) to obtain a high-definition image with the smallest pixel deviation from the image contents at each depth level. Finally, multiple high-definition images may be fused to obtain a target image that matches the first image (i.e., the target image may be used as the registration result of the first image), thereby reducing the pixel deviation between the registration result and the first image.

[0013] In some embodiments, the method of dividing multiple target pixel points whose depth value difference between any two pixel points is within a preset range into a first pixel point set includes: clustering the depth values ​​of the M pixel points through a clustering algorithm to obtain N cluster centers; for each cluster center in the N clusters, dividing multiple pixel points whose depth value difference with the cluster center is within a preset range into a first pixel point set.

[0014] In an embodiment of the present application, the depth values ​​of all pixels in the depth information can be used as a sample set; then the number of cluster centers can be set to N (such as N=2), and N cluster centers can be initialized to be randomly generated (that is, the depth values ​​of N pixels are selected, and the difference between the depth values ​​of these N pixels is large); next, the maximum number of iterations K can be set, and in K iterations, the N best cluster centers are found, and then the depth value of each pixel is compared with the difference of the N best cluster centers respectively, so that the pixels at the same depth level are divided into one pixel set; finally, the image content at different depth levels can be obtained based on the color information of the pixels in each pixel set.

[0015] In some embodiments, matching a second image for each of the first depth images from a target gallery according to a shooting angle and a shooting content to obtain N second images includes: extracting feature points of each of the N first depth images, and comparing them with multiple images in the target gallery respectively; determining the image in the target gallery with the highest matching degree with the feature points of each of the first depth images as the matched second image, to obtain N second images.

[0016] In the embodiment of the present application, based on the corresponding image content at different depth levels and multiple high-definition images (provided by the target library), the image content at different depth levels and multiple high-definition images can be registered respectively. The traditional feature extraction algorithm (Oriented FAST and Rotated BRIEF, ORB), scale invariant feature transformation (Scale Invariant Feature Transform, SIFT), key point detection combined with the traditional image registration method of homography matrix (homography) can be used, and the deep learning registration algorithm such as superpoint combined with superglue can also be used to obtain the high-definition registration result with the best registration effect at different depth levels. Thereby reducing the pixel deviation between the registration result and the first image.

[0017] In some embodiments, the method further includes: using the target image as a reference frame to restore details of the first image.

[0018] In an embodiment of the present application, a target image that matches the first image may be used as a high-definition reference frame to restore details of the first image, thereby improving the picture clarity of the first image.

[0019] In some embodiments, the method further includes: displaying a second user interface on the display screen, wherein a third image is displayed on the second user interface, and the third image is an image obtained by restoring details of the first image based on the target image.

[0020] In an embodiment of the present application, in order to present a higher definition image to the user, the image after detail restoration can be displayed on the display screen of the electronic device, thereby improving the user experience.

[0021] In a second aspect, an embodiment of the present application provides an electronic device, which includes a first processing unit, and the first processing unit is used to: determine depth information of a first image, the depth information including M pixels on the first image, and a depth value of each of the M pixels, where M is an integer greater than 1; extract N first depth images from the first image based on the depth information, wherein each of the N first depth images includes a corresponding first pixel set, the first pixel set includes multiple target pixels in the first image, and the difference in depth values ​​between the multiple target pixels is within a preset range, and N is an integer greater than 1; match a second image for each of the first depth images from a target gallery according to a shooting angle and a shooting content, to obtain N second images; and fuse the N second images to obtain a target image.

[0022] In some embodiments, the device also includes: a first acquisition unit, used to acquire the first image, and estimate the depth values ​​of the M pixels on the first image through a preset algorithm to obtain the depth information, or acquire the depth information of the first image through a sensor.

[0023] In some embodiments, the device also includes: a second processing unit, used to start the camera module of the electronic device and display a first user interface on the display screen, wherein the first user interface displays a captured picture; a first user operation is detected, and in response to the first user operation, the captured picture currently in the first user interface is determined as the first image.

[0024] In some embodiments, the first processing unit is specifically used to: based on the depth values ​​of each of the M pixels, divide multiple target pixels whose depth value difference between any two pixels is within a preset range into a first pixel point set to obtain N first pixel point sets; based on the N first pixel point sets, obtain the N first depth images.

[0025] In some embodiments, the first processing unit is specifically used to: cluster the depth values ​​of the M pixels through a clustering algorithm to obtain N cluster centers; for each cluster center in the N clusters, divide multiple pixel points whose depth value difference with the cluster center is within a preset range into a first pixel point set.

[0026] In some embodiments, the first processing unit is specifically used to: extract feature points of each of the N first depth images, and compare them with multiple images in the target gallery respectively; determine the image in the target gallery with the highest matching degree with the feature points of each of the first depth images as the matched second image, to obtain N second images.

[0027] In some embodiments, the device further includes: a third processing unit, configured to restore details of the first image using the target image as a reference frame.

[0028] In some embodiments, the device further includes: a fourth processing unit, configured to display a second user interface on the display screen, wherein a third image is displayed on the second user interface, and the third image is an image obtained by restoring details of the first image based on the target image.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, which includes a touch screen, a memory, one or more processors, multiple applications, and one or more programs; wherein the one or more programs are stored in the memory, and when the one or more processors execute the one or more programs, the electronic device implements any possible implementation method as in the first aspect.

[0030] In a fourth aspect, a photo preview device is also provided, which has the function of implementing the electronic device behavior in the above method. The above function can be implemented by hardware, or by hardware executing corresponding software. The above hardware or software includes one or more modules corresponding to the above function.

[0031] In a fifth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements any possible implementation method in the first aspect.

[0032] In a sixth aspect, a computer program product comprising instructions is provided, characterized in that when the computer program product is run on an electronic device, the electronic device is enabled to execute any possible implementation method as in the first aspect.

[0033] In a seventh aspect, a computer-readable storage medium is provided, comprising instructions, characterized in that when the instructions are executed on an electronic device, the electronic device executes any possible implementation method in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1A A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application.

[0035] Figure 1B A software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0036] Figure 2A A schematic diagram of an exemplary user interface 21 for an application menu on an electronic device 100 provided in an embodiment of the present application.

[0037] Figure 2B A schematic diagram of the back structure of an electronic device 100 provided in an embodiment of the present application.

[0038] Figure 3A An exemplary schematic diagram of starting image shooting on an electronic device 100 provided in an embodiment of the present application.

[0039] Figure 3B A schematic diagram of an exemplary user interface 31 for image capture on an electronic device 100 provided in an embodiment of the present application.

[0040] Figure 3C An exemplary schematic diagram of image capture on an electronic device 100 provided in an embodiment of the present application.

[0041] Figure 4 A schematic diagram of a depth map provided in an embodiment of the present application.

[0042] Figure 5 A schematic diagram of generating image contents at different depth levels provided in an embodiment of the present application.

[0043] Fig. 6A A schematic diagram of an image registration process provided in an embodiment of the present application.

[0044] Figure 6B A schematic diagram of another image registration process provided in an embodiment of the present application.

[0045] Fig. 7A A schematic diagram of image fusion provided in an embodiment of the present application.

[0046] Figure 7B A schematic diagram of another image fusion provided in an embodiment of the present application.

[0047] Figure 8 A flowchart of an image registration method provided in an embodiment of the present application.

[0048] Fig. 9 A flowchart of another image registration method provided in an embodiment of the present application.

[0049] Fig.10 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0051] The following describes an electronic device, a user interface for such an electronic device, and embodiments for using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to devices equipped with Or a portable electronic device with other operating systems. The portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface or a touch panel. It should also be understood that in some other embodiments, the electronic device may not be a portable electronic device, but a desktop computer with a touch-sensitive surface or a touch panel.

[0052] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (GTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, GTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0053] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0054] First, an exemplary electronic device 100 provided in the following embodiments of the present application is introduced.

[0055] Figure 1A A schematic structural diagram of an electronic device 100 is shown.

[0056] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a 3D camera module 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0057] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0058] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a modem processor, an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, etc. Among them, different processing units may be independent devices or integrated in one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.

[0059] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0060] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100.

[0061] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0062] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the 3D camera module 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0063] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0064] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0065] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0066] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the 3D camera module 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the 3D camera module 193 communicate via the CSI interface to realize the camera function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0067] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the 3D camera module 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0068] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0069] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0070] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0071] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the 3D camera module 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0072] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0073] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0074] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0075] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0076] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2. Exemplarily, the wireless communication module 160 may include a Bluetooth module, a Wi-Fi module, etc.

[0077] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0078] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.

[0079] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0080] The electronic device 100 can realize the camera function through a 3D camera module 193, an ISP, a video codec, a GPU, a display screen 194, an application processor AP, a neural network processor NPU, etc.

[0081] The 3D camera module 193 can be used to collect color image data and depth data of the photographed object. The ISP can be used to process the color image data collected by the 3D camera module 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the 3D camera module 193.

[0082] In some embodiments, the 3D camera module 193 may be composed of a color camera module and a 3D sensing module.

[0083] In some embodiments, the photosensitive element of the camera of the color camera module can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0084] In some embodiments, the 3D sensing module may be a time of flight (TOF) 3D sensing module or a structured light 3D sensing module. Among them, structured light 3D sensing is an active depth sensing technology, and the basic components of the structured light 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of the structured light 3D sensing module is to first emit a light spot of a specific pattern to the object being photographed, and then receive the light spot pattern coding (light coding) on ​​the surface of the object, and then compare the similarities and differences with the original projected light spot, and use the triangulation principle to calculate the three-dimensional coordinates of the object. The three-dimensional coordinates include the distance between the electronic device 100 and the object being photographed. Among them, TOF 3D sensing is also an active depth sensing technology, and the basic components of the TOF 3D sensing module may include an infrared emitter, an IR camera module, etc. The working principle of the TOF 3D sensing module is to calculate the distance (i.e., depth) between the TOF 3D sensing module and the object being photographed by the time when the infrared rays return to obtain a 3D depth map.

[0085] The structured light 3D sensing module can also be applied to face recognition, somatosensory game consoles, industrial machine vision inspection and other fields. The TOF 3D sensing module can also be applied to game consoles, augmented reality (AR) / virtual reality (VR) and other fields.

[0086] In other embodiments, the 3D camera module 193 may also be composed of two or more cameras. The two or more cameras may include a color camera, which can be used to collect color image data of the photographed object. The two or more cameras may use stereo vision technology to collect depth data of the photographed object. Stereo vision technology is based on the principle of human eye parallax. Under natural light, two or more cameras are used to capture images of the same object from different angles, and then triangulation and other operations are performed to obtain the distance information between the electronic device 100 and the photographed object, that is, the depth information.

[0087] In some embodiments, the electronic device 100 may include one or N 3D camera modules 193, where N is a positive integer greater than 1. Specifically, the electronic device 100 may include one front 3D camera module 193 and one rear 3D camera module 193. The front 3D camera module 193 may generally be used to collect color image data and depth data of the photographer facing the display screen 194, and the rear 3D camera module may be used to collect color image data and depth data of the subject (such as a person, scenery, etc.) facing the photographer.

[0088] In some embodiments, the CPU or GPU or NPU in the processor 110 can process the depth data collected by the 3D camera module 193. In some embodiments, the NPU can identify the color image data collected by the 3D camera module 193 (specifically, it can be a color camera module) through a neural network algorithm based on object recognition technology, such as a convolutional neural network algorithm (CNN), to determine the target object being photographed. The CPU or GPU can also be used to run a neural network algorithm to determine the target object being photographed based on the color image data. In some embodiments, the CPU or GPU or NPU can also be used to analyze and process the depth data collected by the 3D camera module 193 (specifically, it can be a 3D sensing module) to obtain image contents at different depth levels, and further, image registration can be performed for the image contents at different depth levels, so that a high-definition image can be matched from the high-definition library for the image contents at each depth level, and then these multiple high-definition images can be fused to obtain a target image with a smaller pixel deviation from the image taken by the 3D camera module 193, and then the target image is used as a high-definition reference frame to restore the details of the image taken by the 3D camera module 193, so that the image taken by the 3D camera module 193 can be repaired more accurately. How to perform image registration based on the depth data of the image will be described in detail in the subsequent embodiments, which will not be repeated here.

[0089] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0090] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-3, MPEG-4, etc.

[0091] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0092] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, data such as music, photos, and videos can be stored in the external memory card.

[0093] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can run the above instructions stored in the internal memory 121, so that the electronic device 100 performs the image registration method of the electronic device provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0094] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0095] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0096] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0097] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0098] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to realize collection of sound signals, noise reduction, identification of sound source, realization of directional recording function, etc.

[0099] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0100] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0101] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0102] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0103] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0104] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0105] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0106] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0107] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0108] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0109] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0110] The touch sensor 180K may also be referred to as a touch panel or a touch-sensitive surface. The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor may transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations may be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, at a position different from that of the display screen 194.

[0111] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0112] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0113] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0114] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0115] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0116] Figure 1A The electronic device 100 shown in the example can display the various user interfaces described in the following embodiments through the display screen 194. The electronic device 100 can detect touch operations in various user interfaces through the touch sensor 180K, such as click operations in various user interfaces (such as touch operations on icons, double-click operations), and for example, upward or downward sliding operations in various user interfaces, or operations of performing circle gestures, etc. In some embodiments, the electronic device 100 can detect motion gestures performed by the user holding the electronic device 100, such as shaking the electronic device, through a gyroscope sensor 180B, an acceleration sensor 180E, etc. In some embodiments, the electronic device 100 can detect non-touch gesture operations through a 3D camera module 193 (such as a 3D camera, a depth camera).

[0117] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0118] Figure 1B It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0119] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0120] The application layer can include a series of application packages.

[0121] like Figure 1B As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0122] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0123] like Figure 1B As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0124] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0125] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0126] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0127] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0128] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0129] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0130] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0131] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0132] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0133] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0134] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0135] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, G.264, MP3, AAC, AMR, JPG, PNG, etc.

[0136] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0137] A 2D graphics engine is a drawing engine for 2D drawings.

[0138] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0139] Figure 1B The software system shown involves application presentation using sharing capabilities (such as gallery, file manager), an instant sharing module that provides sharing capabilities, a print service (print service) and a print spooler that provide printing capabilities, and an application framework layer that provides a printing framework, WLAN services, and Bluetooth services, as well as the kernel and bottom layers that provide WLAN Bluetooth capabilities and basic communication protocols.

[0140] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0141] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch operation and the control corresponding to the touch operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the 3D camera module 193.

[0142] An exemplary user interface for an application menu on electronic device 100 is described below.

[0143] Figure 2A An exemplary user interface 21 for an application menu on the electronic device 100 is shown. Figure 2A As shown, the electronic device 100 may be configured with a 3D camera module 193. In some embodiments, 193-1 may be a color camera, and 193-2 may be a structured light 3D camera module. In other embodiments, 193-1 may be a color camera, and 193-2 may be a TOF 3D camera module. In some embodiments, 193-1 and 193-2 may be two color cameras. Figure 2A As shown, the 3D camera module 193 can be disposed at the top of the electronic device 100, such as the "bangs" position of the electronic device 100 (i.e. Figure 2A It can be known that in addition to the 3D camera module 193, the area AA may also include an illuminator 197 (not shown in FIG. Figure 1A ), speaker 170A, proximity light sensor 180G, ambient light sensor 180L, etc. In some embodiments, as shown in Figure 2B As shown, the back of the electronic device 100 may also be configured with a 3D camera module 193 and an illuminator 197 .

[0144] like Figure 2A As shown, the user interface 21 may include: a status bar 201, a tray with common application icons, a calendar indicator, a weather indicator, a navigation bar, and other application icons, etc. Among them:

[0145] The status bar 201 may include: one or more signal strength indicators 201-1 of a mobile communication signal (also referred to as a cellular signal), an indicator 201-2 of the operator of the mobile communication signal, a time indicator 201-3, a battery status indicator 201-4, and the like.

[0146] Calendar indicators may be used to indicate the current time, such as date, day of the week, hour and minute information, etc.

[0147] The weather indicator can be used to indicate the weather type, such as cloudy to sunny, light rain, etc., and can also be used to indicate information such as temperature.

[0148] The tray with common application icons may display: a camera icon 218 , an address book icon 219 , a phone icon 220 , a short message icon 221 , and the like.

[0149] The navigation bar may include system navigation keys such as a back button, a home screen button, and an outgoing task history button. When it is detected that the user clicks the back button, the electronic device 100 may display the previous page of the current page. When it is detected that the user clicks the home screen button, the electronic device 100 may display the home screen. When it is detected that the user clicks the outgoing task history button, the electronic device 100 may display the tasks that the user has recently opened. The naming of each navigation key may also be other, and this application does not impose any restrictions on this. Not limited to virtual buttons, each navigation key in the navigation bar may also be implemented as a physical button.

[0150] Other application icons may include, for example, an alarm clock icon 202, a calendar icon 203, a gallery icon 204, a memo icon 205, a file management icon 206, an email icon 207, a music icon 208, a calculator icon 209, a Huawei video icon 210, a sports health icon 211, a weather icon 212, a browser icon 213, a smart life icon 214, a settings icon 215, a recorder icon 216, and an application store icon 217. The user interface 21 may also include a page indicator 222. Other application icons may be distributed on multiple pages, and the page indicator 222 may be used to indicate which page the user is currently browsing. The user may slide the area of ​​other application icons left and right to browse application icons on other pages.

[0151] In some embodiments, Figure 2A The user interface 21 exemplarily shown may be a main interface (Gome screen).

[0152] In some other embodiments, the electronic device 100 may further include a home screen key. The home screen key may be a physical key or a virtual key. The home screen key may be used to receive instructions from the user and return the currently displayed UI to the main interface, so that the user can view the home screen at any time. The above instructions may specifically be an operation instruction for the user to press the home screen key once, or an operation instruction for the user to press the home screen key twice in a short period of time, or an operation instruction for the user to long press the home screen key within a predetermined time. In some other embodiments of the present application, the home screen key may also be integrated with a fingerprint identifier so that fingerprint collection and identification can be performed when the home screen key is pressed.

[0153] Understandably, Figure 2A The user interface on the electronic device 100 is merely illustrated as an example and should not constitute a limitation on the embodiments of the present application.

[0154] An application scenario involved in the present application will be described below: an image capturing scenario.

[0155] like Figure 3A As shown, the electronic device 100 can detect a touch operation on the camera icon 218 (such as a click operation on the icon 218), and in response to the operation, a Figure 3B The user interface 31 is shown as an example. The user interface 31 may be a user interface of a "camera" application, which can be used by a user to take pictures, such as taking photos and videos. "Camera" is an image-taking application on electronic devices such as smartphones and tablets. This application does not limit the name of the application. In other words, a user can Figure 3A Clicking icon 218 on the user interface 21 shown in the figure opens the “camera” user interface 31. Not limited to this, the user can also open the user interface 31 in other applications, for example, the user clicks the shooting control in “WeChat” to open the user interface 31. “WeChat” is a social application that supports users to share photos taken with others.

[0156] Figure 3B A user interface 31 of a "camera" application on an electronic device such as a smart phone is exemplarily shown. Figure 3B As shown, the user interface 31 may include: area 301, shooting mode list 302, control 303, control 304 and control 305. Among them:

[0157] Area 301 may be referred to as a preview frame 301. The preview frame 301 may be used to display a color image captured in real time by the 3D camera module 193. The electronic device may refresh the display content therein in real time so that the user can preview the color image currently captured by the 3D camera module 193. Here, the 3D camera module 193 may be a rear camera or a front camera.

[0158] One or more shooting mode options may be displayed in the shooting mode list 302. The one or more shooting options may include: night scene mode option 302A, portrait mode option 302B, photo mode option 302C, video mode option 302D and more shooting mode options 302E. The one or more shooting options may be displayed as text information on the interface, for example, the night scene mode option 302A, portrait mode option 302B, photo mode option 302C, video mode option 302D and more shooting mode options 302E may correspond to the text "night scene", "portrait", "photo", "video", "more" respectively. Not limited to this, the one or more shooting options may also be displayed as icons or other forms of interactive elements (IE) on the interface. In some embodiments, the electronic device 100 may select the photo mode option 302C by default, and the display state of the photo mode option 302C (such as the photo mode option 302C being highlighted) may indicate that the photo mode option 302C has been selected.

[0159] The electronic device 100 can detect a user operation on a shooting mode option, and the user operation can be used to select a shooting mode. In response to the operation, the electronic device 100 can start the shooting mode selected by the user. In particular, when the user operation acts on more shooting mode options 302E, the electronic device 100 can further display more other shooting mode options, such as a large aperture shooting mode option, a slow motion shooting mode option, etc., and can show the user more abundant camera functions. Figure 3B As shown, more shooting mode options 302E may not be displayed in the shooting mode list 302 , and the user may browse other shooting mode options by sliding left / right in the shooting mode list 302 .

[0160] Control 303 can be used to monitor the user operation that triggers shooting (taking pictures or recording videos). Figure 3C As shown, when the electronic device detects a user operation acting on the control 303 (such as a click operation on the control 303), in response to the operation, the electronic device 100 can save the image in the preview box 301. The saved image can be a picture or a video. In addition, the electronic device 100 can also display a thumbnail of the saved image in the control 304. In other words, the user can click on the control 303 to trigger shooting. Among them, the control 303 can be a button or other forms of controls. In this application, the control 303 can be referred to as a shooting control. It should be noted that an image registration method provided in an embodiment of the present application can be used to match the image captured by the electronic device 100 to a high-definition image with a small pixel deviation, and then the image captured by the electronic device 100 can be restored in detail based on the high-definition image with a small pixel deviation, so as to improve the clarity of the image captured by the electronic device. An image registration method provided in the present application will be described in detail in subsequent embodiments, and will not be described in detail here.

[0161] The control 305 can be used to monitor the user operation that triggers the camera switching. The electronic device 100 can detect the user operation acting on the control 305 (such as a click operation on the control 305), and in response to the operation, the electronic device 100 can switch the camera (such as switching the rear camera to the front camera, or switching the front camera to the rear camera).

[0162] Control 304 can be used to monitor user operations that trigger the opening of the "Gallery". The electronic device 100 can detect user operations acting on the control 304 (such as a click operation on the control 304). In response to the operation, the electronic device 100 can display the user interface of the "Gallery", in which pictures saved by the electronic device 100 can be displayed. It should be noted that in the embodiment of the present application, the images in the "Gallery" can be images after detail restoration based on high-definition images, and the clarity is higher than that of the original images taken by the electronic device 100. Here, the "Gallery" is an application for picture management on electronic devices such as smart phones and tablets, which can also be called an "album". This embodiment does not limit the name of the application. The "Gallery" can support users to perform various operations on pictures stored on electronic devices, such as browsing, editing, deleting, selecting, and the like.

[0163] In the prior art, registration is performed only based on the planar feature points of the low-definition image, so as to select an image with the highest matching degree with the planar feature points of the low-definition image from the high-definition image library as the registration result. However, in this process, the visual differences between different objects in the low-definition image are ignored, resulting in a large pixel deviation between the registration result and the low-definition image. In the embodiment of the present application, considering the visual differences between the image contents at different depth levels of the low-definition image, image registration is performed for the image contents at each depth level of the low-definition image, and the multiple registration results are fused to obtain a high-definition image that matches the low-definition image (i.e., the fused high-definition image is used as the registration result of the first image), thereby reducing the pixel deviation between the registration result and the first image.

[0164] The following will be combined Figure 1A The structure of the electronic device 100 shown, as well as the application scenarios, describe the image registration method provided in the present application from the following points: how to determine the first image (which can be understood as the image to be registered); how to obtain image contents at different depth levels based on the first image; how to match high-definition images for image contents at different depth levels; how to determine the registration result of the first image (i.e., the target image).

[0165] Aspect 1: How to determine the first image.

[0166] Combined with the above Figure 3A-3C The image capture scene described, such as Figure 3A As shown, a user interface 21 is displayed on the electronic device, and when a touch operation of a user on a camera icon 218 is detected, the camera module of the electronic device (such as the above-mentioned Figure 1A The 3D camera module 193 in the electronic device is displayed as follows Figure 3B The user interface 31 shown (which may be referred to as the first user interface) displays a captured image on the first user interface; when the electronic device detects a user operation that triggers shooting (such as a click operation on the control 303, which may be referred to as the first user operation), in response to the first user operation, the electronic device may determine the captured image in the current first user interface as the first image, and illustratively, the first image may be an RGB image. It should be noted that due to the imaging limitations of the camera module, when the electronic device is in a high-magnification shooting scenario, such as when the electronic device is shooting a distant scene, the first image captured may have problems such as blurred details, and the first image may be restored in detail through software to improve the clarity of the first image.

[0167] Aspect 2: How to obtain image contents at different depth levels based on the first image.

[0168] Point 1: Determine the depth information of the first image.

[0169] Specifically, the depth information of the first image may include M pixels on the first image, and the depth value of each of the M pixels, where M is an integer greater than 1. The depth value may represent the distance between a point in the scene and the camera module, and may be a value between 0 and 255. Optionally, a depth map of the first image may be generated based on the depth information of the first image. Assuming that the first image includes M pixels, a depth map of the first image may be generated based on the depth value of each pixel. Since the depth value of the pixel may represent the distance between a point in the scene and the camera module, the depth map of the first image contains information related to the distance from the surface of the scene object to the viewpoint, reflects the distance from the object to the plane of the camera module, and also reflects the geometry of the visible surface of the object. For example, Figure 4 As shown, Figure 4 A schematic diagram of a depth map provided for an embodiment of the present application, in which the farther the object is from the camera in the depth map, the darker the color. The depth map of the first image in the figure includes two target objects, namely plants and buildings. Through the depth map, it can be seen that plants and buildings are at different depth levels, and plants are closer to the camera, and buildings are farther away from the camera.

[0170] In some embodiments, after acquiring the first image, the depth value of M pixels on the first image can be estimated by a preset algorithm to obtain depth information. Specifically, the preset algorithm may include but is not limited to a depth estimation network, a convolutional neural network (CNN), a recurrent neural network (RNN), an adversarial neural network, and the like. Exemplarily, the preset algorithm is used as a depth estimation network for illustration. The input of the depth estimation network can be a low-definition mobile phone image taken by the user (i.e., the first image). The depth estimation network can estimate the depth value of each pixel on the image to obtain the depth information of the image (i.e., obtain the depth value of each pixel). Optionally, a depth map of the first image can be generated based on the estimated depth information.

[0171] In some embodiments, when shooting the first image, if the shooting device has a depth sensor, the depth information of the first image can be obtained through the depth sensor. Optionally, a depth map of the first image can be generated based on the measured depth information.

[0172] Point 2: Based on the depth information, determine the image content at different depth levels.

[0173] (1) Determine N first pixel point sets (N is an integer greater than 1).

[0174] Specifically, the first image may include image contents at N depth levels. The image contents at one depth level may include multiple target pixels. Therefore, the pixels at the same depth level may be divided into a set as much as possible. Then, N images at different depth levels may be obtained based on the N pixel sets.

[0175] In some embodiments, based on the depth values ​​of the respective M pixels, a plurality of target pixels whose depth value difference between any two pixels is within a preset range are divided into a first pixel set, and N first pixel sets are obtained. Specifically, assuming that the depth value of each pixel is between 0 and 255, the number N of first pixel sets can be determined first according to actual conditions, and then the difference in depth value between any two pixels can be calculated, and then a plurality of target pixels whose depth value difference between any two pixels is within a preset range can be divided into a first pixel set, and finally the pixels at the same depth level are divided into a first pixel set.

[0176] In some embodiments, the depth values ​​of M pixels are clustered by a clustering algorithm to obtain N cluster centers; for each cluster center in the N clusters, multiple pixels whose depth value difference with the cluster center is within a preset range are divided into a first pixel set. Specifically, assuming that the depth value of each pixel in the depth information is between 0 and 255, the depth values ​​of all pixels in the depth information can be used as a sample set; the number of cluster centers can be set to N (such as N = 2), and N cluster centers are randomly generated (that is, the depth values ​​of N pixels are selected, and the difference between the depth values ​​of these N pixels is large); the maximum number of iterations K is set; in K iterations, N optimal cluster centers are found to meet the minimum cumulative difference between the depth value of each pixel and the cluster center to which it belongs; the depth value of each pixel is compared with the difference of the N optimal cluster centers respectively, and the cluster center to which each pixel belongs is obtained, and all pixels under the cluster center are divided into the same pixel set, and finally the pixels at the same depth level are divided into a first pixel set.

[0177] (2) Obtain N first depth images based on the N first pixel point sets.

[0178] Specifically, each first depth image may include a corresponding first pixel set; each of the N first pixel sets may include multiple target pixels at the same depth level, and based on the color information of the target pixels in each set, the first depth image corresponding to the set may be obtained. It should be noted that each first depth image has the same image size as the first image (i.e., the same length and width), and the pixel positions on the first depth image correspond one-to-one to the pixel positions on the first image. The depth level of each pixel of the first image is the same as the depth level of the pixel at the corresponding depth image at that position, so the corresponding first image content at different depth levels may be obtained. The other pixels on the first depth image except the target pixels can be filled with three-channel black pixels (0, 0, 0), and finally N first depth images are obtained.

[0179] For example, Figure 5 As shown is a schematic diagram of generating image content at different depth levels provided by an embodiment of the present application. In the figure, a depth map of the first image can be generated based on the depth information of the first image. It can be seen from the depth map that the first image can be divided into two depth levels, namely the depth level where the plant is located and the depth level where the building is located. Further, two pixel point sets can be obtained based on the depth information of the first image, corresponding to the plant and the building respectively, and then two hierarchical depth maps can be generated based on the two pixel point sets, and then the color information of the target pixel point in the hierarchical depth map is obtained to obtain the first depth image, such as Figure 5 In the method, two first depth images are generated based on the first image, which are an image of the depth where the plant is located and an image of the depth where the building is located.

[0180] Aspect three: How to match high-definition images for image content at different depth levels.

[0181] Point 1: Determine the target gallery.

[0182] Specifically, the target gallery may be a high-definition gallery stored locally in the electronic device, or may be a high-definition gallery stored in a cloud database. Optionally, the electronic device may obtain the shooting location information of the first image, and determine multiple high-definition images in the target gallery based on the shooting location information. The multiple high-definition images may be images with the same shooting environment as the first image but with different shooting angles.

[0183] Point 2: Match a second image from the target gallery for each first depth image.

[0184] Specifically, the second image can be understood as a high-definition image with a small pixel deviation selected from the target image library according to the shooting content and shooting angle of the first image.

[0185] In some embodiments, feature points of each of the N first depth images can be extracted and compared with multiple images in the target gallery; the image in the target gallery with the highest degree of match with the feature points of each first depth image is determined as the matched second image to obtain N second images. Specifically, the mobile phone image content at different depth levels can be registered with the multiple high-definition images according to the corresponding mobile phone image content at different depth levels (i.e., the first depth image) and multiple high-definition images (provided by the target gallery). Traditional feature extraction algorithms (Oriented FAST and Rotated BRIEF, ORB), scale invariant feature transform (Scale Invariant Feature Transform, SIFT), key point detection combined with homography matrix (homography) traditional image registration methods can be used, and deep learning registration algorithms such as superpoint combined with superglue can also be used to obtain high-definition registration results with the best registration effect at different depth levels.

[0186] For example, Fig. 6A As shown, Fig. 6A A schematic diagram of an image registration process provided in an embodiment of the present application, when registering a first depth image of a plant, multiple feature points of the plant in the first depth image can be first extracted, and then based on the multiple feature points and multiple high-definition images in the high-definition image library ( Fig. 6A Only 6 high-definition images are shown as examples in the figure) for comparison, and then a high-definition image (b) is matched from the high-definition image library, and the high-definition image (b) is used as the registration result of the first depth image of the plant. By comparing the first depth image and the registration result (i.e., the second image), it can be found that the image taken by the electronic device has a phenomenon of detail loss, such as details such as branches in the plant.

[0187] Next, if Figure 6B As shown, Figure 6B Another image registration process schematic diagram provided in an embodiment of the present application, when registering a first depth image of a building, firstly, a plurality of feature points of the building in the first depth image can be extracted, and then based on the plurality of feature points and a plurality of high-definition images in the high-definition image library ( Figure 6B Only 6 high-definition images are shown as examples in the figure) for comparison, and then a high-definition image (e) is matched from the high-definition image library, and the high-definition image (e) is used as the registration result of the first depth image of the building. By comparing the first depth image and the registration result (i.e., the second image), it can be found that the image taken by the electronic device has lost details, such as the windows on the building.

[0188] Finally, a second image is matched for each first depth image in the target image library.

[0189] Aspect 4: How to determine the registration result of the first image (ie, the target image).

[0190] Specifically, multiple second images (i.e., high-definition images matching image contents at different depth levels) can be fused to obtain a target image. Since the content and angle of each second image have a small pixel deviation from the corresponding first depth image, the pixel deviation between the target image obtained after fusing the registration results of images at different depths and the first image is also relatively small.

[0191] Optionally, based on N high-definition registration images at different depth levels, the high-definition registration images are consistent with the size of the mobile phone image, and the pixel points at the same position of each of the N high-definition registration image results can be added to achieve the fusion of the N high-definition registration image results to obtain the final registration result (i.e., the target image). Fig. 7A As shown, Fig. 7A A schematic diagram of image fusion provided in an embodiment of the present application, wherein an addition operation is performed on each pixel point at the same position of the high-definition image (b) and the high-definition image (e) to obtain a target image.

[0192] Optionally, the target object in each second target image may be extracted first, and then multiple target objects may be fused to obtain the target image. Figure 7B The figure is a schematic diagram of another image fusion provided in an embodiment of the present application, which can first extract the target object in each second image, and then add the pixel points at the same position of multiple target object images to achieve the fusion of multiple target object images to obtain the target image.

[0193] Aspect five: restore details of the first image based on the target image.

[0194] Specifically, the target image can be used as a high-definition reference frame to restore the details of the first image to obtain a third image, such as restoring the details of buildings and plants based on the target image, thereby improving the picture clarity of the first image.

[0195] Aspect six: displaying a third image on the electronic device.

[0196] Specifically, a second user interface is displayed on a display screen of the electronic device, and a third image is displayed on the second user interface. The third image is an image obtained by restoring details of the first image based on the target image, and the third image has higher definition than the first image.

[0197] In an embodiment of the present application, taking into account the visual deviation of image contents at different depth levels, image contents at different depth levels can be extracted based on the first image, and image registration is performed for the image contents at each depth level (i.e., a high-definition image is matched from the gallery for the image contents at each depth level according to the shooting content and the shooting angle) to obtain a high-definition image with the smallest pixel deviation from the image contents at each depth level. Finally, multiple high-definition images are fused to obtain a target image that matches the first image (i.e., the target image is used as the registration result of the first image), thereby reducing the pixel deviation between the registration result and the first image.

[0198] The following takes a smartphone as an example of an electronic device and describes the image registration method in this application in combination with actual application scenarios.

[0199] S401: The smartphone determines a first image.

[0200] Among them, the relevant description of the smart phone can be found in the above Figure 1A and Figure 1B The description of the electronic device 100 in FIG. 1 is not repeated here.

[0201] Among them, the specific implementation method of the smart phone determining the first image can refer to the relevant description in the above aspect one, which will not be repeated here.

[0202] S402: The smart phone determines a registration result of the first image (target image).

[0203] Specifically, the smartphone determines the depth information of the first image; the smartphone extracts N first depth images from the first image based on the depth information; the smartphone matches a second image for each first depth image from the target gallery according to the shooting angle and the shooting content, and obtains N second images; the smartphone fuses the N second images to obtain the target image. It should be noted that the target gallery can be stored locally on the smartphone or in a cloud database.

[0204] Among them, the specific implementation of the smart phone determining the depth information of the first image and extracting N first depth images from the first image based on the depth information can be found in the relevant description in the above-mentioned aspect 2, which will not be repeated here.

[0205] Among them, regarding the specific implementation method of the smartphone matching a second image for each first depth image from the target gallery according to the shooting angle and the shooting content to obtain N second images, please refer to the relevant description in the above aspect three, which will not be repeated here.

[0206] Among them, the specific implementation method of a smart phone fusing N second images to obtain a target image can be found in the relevant description in the above aspect 4, which will not be repeated here.

[0207] S403: The smart phone restores details of the first image based on the target image.

[0208] Among them, the specific implementation method of the smart phone restoring the details of the first image can refer to the relevant description in the above aspect five, which will not be repeated here.

[0209] In some embodiments, the smart phone may display a third image on the display screen. For details, please refer to the relevant description in the above aspect six, which will not be repeated here.

[0210] The following takes the electronic device as a cloud server as an example, and describes the image registration method in this application in combination with actual application scenarios.

[0211] S501: The cloud server determines a first image.

[0212] In some embodiments, the cloud server may receive an image captured by a target device (eg, a smart phone) and use it as the first image.

[0213] S502: The cloud server determines the registration result of the first image (target image).

[0214] Among them, the method for the cloud server to determine the registration result of the first image can be consistent with the method for the above-mentioned smart phone to determine the registration result of the first image, which will not be repeated here.

[0215] S503: The cloud server restores details of the first image based on the target image.

[0216] Among them, the method for the cloud server to restore the details of the first image based on the target image can be consistent with the method for the above-mentioned smart phone to restore the details of the first image based on the target image, which will not be repeated here.

[0217] Next, the specific method architecture on which the embodiments of the present application are based is described.

[0218] See also Figure 8 , Figure 8 This is a flow chart of an image registration method in an embodiment of the present application. Figure 8 Based on the above image shooting scene, the image registration method in the embodiment of the present application is described. It should be noted that in order to describe the image registration method in the embodiment of the present application in more detail, the corresponding execution subject described in each process step can be the above Figure 1A The electronic device 100 in the embodiment does not mean that the corresponding method flow can only be performed by the described execution subject in the embodiment of the present application.

[0219] Step S601: the electronic device 100 determines depth information of a first image.

[0220] For the description of the electronic device 100, please refer to the above Figure 1A and Figure 1B The description of the electronic device 100 in FIG. 1 is not repeated here.

[0221] Among them, the specific implementation method of the electronic device 100 determining the above-mentioned first image can refer to the relevant description in the above-mentioned aspect one, and will not be repeated here.

[0222] Among them, the specific implementation method of the electronic device 100 determining the depth information of the above-mentioned first image can refer to the relevant description of point 1 in the above-mentioned aspect 2, and will not be repeated here.

[0223] Step S602: the electronic device 100 extracts N first depth images from the first image based on the depth information.

[0224] Among them, the specific implementation of the electronic device 100 extracting N first depth images from the above-mentioned first image based on the above-mentioned depth information can refer to the relevant description of point 2 in the above-mentioned aspect 2, which will not be repeated here.

[0225] In step S602, combining Fig. 9 , a possible implementation of step S602 is described in detail. For example, Fig. 9 As shown, Fig. 9 A flow chart of another image registration method provided for an embodiment of the present application, in which an image taken by a mobile phone is input as a first image into a depth estimation network to obtain a visible depth map; then, image contents at different depth levels are obtained based on a clustering algorithm; further, based on the image contents at different depth levels and multiple frames of high-definition images (provided by retrieval), optimal registration results at different depth levels are obtained; finally, the optimal registration results at different depth levels are fused to obtain a high-definition image that matches the first image.

[0226] Step S603: the electronic device 100 matches a second image for each first depth image in the target gallery to obtain N second images.

[0227] Among them, the specific implementation method of the electronic device 100 matching the second image for the first depth image from the target gallery can refer to the relevant description in the above aspect three, which will not be repeated here.

[0228] Step S604: the electronic device 100 fuses the N second images to obtain a target image.

[0229] Among them, the specific implementation method of the electronic device 100 obtaining the target image based on the above-mentioned N second images can refer to the relevant description in the above-mentioned aspect 4, which will not be repeated here.

[0230] In some embodiments, the electronic device 100 uses the target image as a reference frame to restore details of the first image.

[0231] Among them, the specific implementation method of the electronic device 100 restoring the details of the above-mentioned first image can refer to the relevant description in the above-mentioned aspect five, and will not be repeated here.

[0232] In some embodiments, a second user interface is displayed on the display screen of the electronic device 100 , and a third image is displayed on the second user interface. The third image is an image obtained by restoring details of the first image based on the target image.

[0233] Among them, the electronic device 100 can display the third image on the display screen. For details, please refer to the relevant description in the above aspect six, which will not be repeated here.

[0234] In an embodiment of the present application, taking into account the visual deviation of image contents at different depth levels, image contents at different depth levels can be extracted based on the first image, and image registration is performed for the image contents at each depth level (i.e., a high-definition image is matched from the gallery for the image contents at each depth level according to the shooting content and the shooting angle) to obtain a high-definition image with the smallest pixel deviation from the image contents at each depth level. Finally, multiple high-definition images are fused to obtain a target image that matches the first image (i.e., the target image is used as the registration result of the first image), thereby reducing the pixel deviation between the registration result and the first image.

[0235] The above describes in detail the method of the embodiment of the present application, and the following provides related equipment of the embodiment of the present application.

[0236] See also Fig.10 , Fig.10 A schematic diagram of an electronic device provided in an embodiment of the present application, the electronic device 70, the device may include a first processing unit 701, a first acquisition unit 702, a second processing unit 703, a third processing unit 704, and a fourth processing unit 705. Among them,

[0237] The first processing unit 701 is used to: determine the depth information of the first image, where the depth information includes M pixels on the first image and the depth value of each of the M pixels, where M is an integer greater than 1; extract N first depth images from the first image based on the depth information, where each of the N first depth images includes a corresponding first pixel set, the first pixel set includes multiple target pixels in the first image, the difference in depth values ​​between the multiple target pixels is within a preset range, and N is an integer greater than 1; match a second image for each first depth image from a target gallery to obtain N second images; and fuse the N second images to obtain a target image.

[0238] Among them, the specific implementation of the first processing unit 701 determining the depth information of the first image and extracting N first depth images from the first image based on the depth information can be found in the relevant description in the above aspect 2, which will not be repeated here.

[0239] Among them, the specific implementation of the first processing unit 701 matching a second image for each first depth image from the target library can be found in the relevant description in the above aspect three, which will not be repeated here.

[0240] Among them, the specific implementation of the first processing unit 701 fusing N second images to obtain the target image can be found in the relevant description in the above aspect 4, which will not be repeated here.

[0241] The first processing unit 701 may be Figure 1A A processing unit in the processor 110 in the electronic device 100.

[0242] In some embodiments, the first acquisition unit 702 is used to acquire a first image, and estimate the depth values ​​of M pixels on the first image through a preset algorithm to obtain depth information, or acquire the depth information of the first image through a sensor.

[0243] Among them, the specific implementation method of the first acquisition unit 702 determining the depth information of the above-mentioned first image can refer to the relevant description of point 1 in the above-mentioned aspect 2, and will not be repeated here.

[0244] The first acquisition unit 702 may be Figure 1A An acquisition unit in the processor 110 in the electronic device 100.

[0245] In some embodiments, the second processing unit 703 is used to start the camera module of the electronic device and display a first user interface on the display screen, where the captured picture is displayed; when the second processing unit 703 detects the first user operation, in response to the first user operation, the captured picture in the current first user interface is determined as the first image.

[0246] The specific implementation of the second processing unit 703 determining the first image can refer to the related description in the above aspect 1, which will not be repeated here.

[0247] The second processing unit 703 may be Figure 1A A processing unit in the processor 110 in the electronic device 100; the camera module can be Figure 1A The 3D camera module 193 in the electronic device 100; the display screen can be Figure 1A The display screen 194 in the electronic device 100.

[0248] In some embodiments, the third processing unit 704 is configured to use the target image as a reference frame to restore details of the first image.

[0249] Among them, the specific implementation method of the third processing unit 704 restoring the details of the above-mentioned first image can refer to the relevant description in the above-mentioned aspect five, and will not be repeated here.

[0250] In some embodiments, the fourth processing unit 705 is used to display a second user interface on the display screen, where a third image is displayed on the second user interface, and the third image is an image obtained by restoring details of the first image based on the target image.

[0251] Among them, the fourth processing unit 705 can display the third image on the display screen. For details, please refer to the relevant description in the above aspect six, which will not be repeated here.

[0252] An embodiment of the present application provides an electronic device, the electronic device includes a processor, the processor is configured to support the electronic device to implement a corresponding function in an image registration method. The electronic device may also include a memory, the memory is used to couple with the processor, and the memory stores the necessary program instructions and data of the electronic device. The electronic device may also include a communication interface for the electronic device to communicate with other devices or a communication network.

[0253] The embodiment of the present application provides a photo preview device, which has the function of realizing the electronic device behavior in the above-mentioned image registration method. The above-mentioned function can be realized by hardware, or by hardware executing corresponding software. The above-mentioned hardware or software includes one or more modules corresponding to the above-mentioned function.

[0254] An embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements any possible implementation of the image registration method described above.

[0255] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute any possible implementation of the image registration method in any possible implementation of the above-mentioned image registration method.

[0256] The present application provides a computer storage medium, characterized in that the computer storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above-mentioned image registration methods is implemented.

[0257] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0259] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0261] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. Among them, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviated: ROM) or random access memory (Random Access Memory, abbreviated: RAM) and other media that can store program codes.

[0262] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image registration method, characterized in that: Applied to electronic equipment, the method comprises: Determine depth information of the first image, where the depth information includes M pixels on the first image and a depth value of each of the M pixels, where M is an integer greater than 1; Extracting N first depth images from the first image based on the depth information, wherein each of the N first depth images includes a corresponding first pixel point set, the first pixel point set includes a plurality of target pixel points in the first image, a difference in depth values ​​between the plurality of target pixel points is within a preset range, and N is an integer greater than 1; Matching a second image to each of the first depth images from the target image library to obtain N second images; The N second images are fused to obtain a target image.

2. The method according to claim 1, characterized in that The method further comprises: The first image is acquired, and depth values ​​of the M pixels on the first image are estimated by a preset algorithm to obtain the depth information, or the depth information of the first image is acquired by a sensor.

3. The method according to claim 1 or 2, characterized in that The electronic device includes a display screen and a camera module, and the method further includes: Starting the camera module and displaying a first user interface on the display screen, wherein the first user interface displays a captured image; A first user operation is detected, and in response to the first user operation, a captured picture currently in the first user interface is determined as the first image.

4. The method according to claim 1, characterized in that The extracting N first depth images from the first image based on the depth information includes: Based on the depth values ​​of the M pixels, a plurality of target pixels whose depth value differences between any two pixels are within a preset range are divided into a first pixel set to obtain N first pixel sets; Based on the N first pixel point sets, the N first depth images are obtained.

5. The method according to claim 4, characterized in that The step of dividing a plurality of target pixel points whose depth value difference between any two pixel points is within a preset range into a first pixel point set includes: Clustering the depth values ​​of the M pixels using a clustering algorithm to obtain N cluster centers; For each cluster center in the N clusters, a plurality of pixel points whose depth value difference with the cluster center is within a preset range are divided into a first pixel point set.

6. The method according to claim 1, characterized in that The matching a second image for each of the first depth images from the target image library to obtain N second images includes: Extracting feature points of each of the N first depth images, and comparing them with the multiple images in the target gallery respectively; An image in the target image library with the highest matching degree with the feature points of each of the first depth images is determined as the matched second image, to obtain N second images.

7. The method according to claim 1, characterized in that The method further comprises: The target image is used as a reference frame to restore details of the first image.

8. The method according to claim 3, characterized in that The method further comprises: A second user interface is displayed on the display screen, and a third image is displayed on the second user interface. The third image is an image obtained by restoring details of the first image based on the target image.

9. An electronic device, characterized in that: The electronic device comprises a first processing unit, wherein the first processing unit is configured to: Determine depth information of the first image, where the depth information includes M pixels on the first image and a depth value of each of the M pixels, where M is an integer greater than 1; Extracting N first depth images from the first image based on the depth information, wherein each of the N first depth images includes a corresponding first pixel point set, the first pixel point set includes a plurality of target pixel points in the first image, a difference in depth values ​​between the plurality of target pixel points is within a preset range, and N is an integer greater than 1; Matching a second image to each of the first depth images from the target image library to obtain N second images; The N second images are fused to obtain a target image.

10. The device according to claim 9, characterized in that The device also includes: The first acquisition unit is used to acquire the first image and estimate the depth values ​​of the M pixels on the first image through a preset algorithm to obtain the depth information, or acquire the depth information of the first image through a sensor.

11. The device according to claim 9 or 10, characterized in that The device also includes: A second processing unit is used to start the camera module of the electronic device and display a first user interface on the display screen, wherein the first user interface displays the captured image; A first user operation is detected, and in response to the first user operation, a captured picture currently in the first user interface is determined as the first image.

12. The device according to claim 9, characterized in that The first processing unit is specifically configured to: Based on the depth values ​​of the M pixels, a plurality of target pixels whose depth value differences between any two pixels are within a preset range are divided into a first pixel set to obtain N first pixel sets; Based on the N first pixel point sets, the N first depth images are obtained.

13. The device according to claim 12, characterized in that The first processing unit is specifically configured to: Clustering the depth values ​​of the M pixels using a clustering algorithm to obtain N cluster centers; For each cluster center in the N clusters, a plurality of pixel points whose depth value difference with the cluster center is within a preset range are divided into a first pixel point set.

14. The device according to claim 9, characterized in that The first processing unit is specifically configured to: Extracting feature points of each of the N first depth images, and comparing them with the multiple images in the target gallery respectively; An image in the target image library with the highest matching degree with the feature points of each of the first depth images is determined as the matched second image, to obtain N second images.

15. The device according to claim 9, characterized in that The device also includes: The third processing unit is used to restore the details of the first image by taking the target image as a reference frame.

16. The device according to claim 11, characterized in that The device also includes: The fourth processing unit is used to display a second user interface on the display screen, wherein the second user interface displays a third image, and the third image is an image obtained by restoring details of the first image based on the target image.

17. An electronic device comprising a touch screen, a memory, one or more processors, a plurality of application programs, and one or more programs; wherein: The one or more programs are stored in the memory; it is characterized in that when the one or more processors execute the one or more programs, the electronic device implements the method as claimed in any one of claims 1 to 8.

18. A computer storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 8.

Images