Image processing method and electronic device

Abstract

The embodiment of the present application provides an image processing method and an electronic device, which are applied to the field of terminal technology, and can enhance the pixel information of an image fused with a reference image, reduce the negative impact of unregistered areas in the image fusion process, thereby improving the problem of image clarity regression after fusion, and making the image processing effect better. In this method, the electronic device transforms the position of the first pixel of the image to be registered according to the position mapping relationship from the reference image to the image to be registered of the first pixel matching the reference image and the image to be registered, and obtains a first registered image. And according to the difference value between the registered object of the first registered image and the registered object of the reference image, the reference weight of the registered object whose difference value meets the first condition in the first registered image is adjusted to obtain a second registered image corresponding to the image to be registered; then the reference image and N second registered images are fused to obtain a target image.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and electronic device. Background Art

[0002] Due to the influence of factors such as the shooting environment, the hardware conditions of the shooting equipment, and image compression, the captured images often have obvious noise, blur, reflections, etc., so it is necessary to perform image quality enhancement processing.

[0003] In the related technology, in order to solve the problems of image noise, blur, reflection, etc., a group of image sequences are obtained, the feature information in the image sequences is correlated and analyzed, and the image sequences are fused into one image based on the exchange of feature information of each image in the image sequence. However, the image fused by the fusion of multiple images may have the problem of loss of clarity. Summary of the invention

[0004] Based on this, the embodiments of the present application provide an image processing method and an electronic device, which can reduce the negative impact of unregistered objects in the image fusion process, thereby improving the problem of image clarity regression after fusion, and achieving better image processing effects.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, in which the electronic device obtains multiple frames of images corresponding to a target object, wherein the multiple frames of images include a reference image and N images to be registered, and N is an integer greater than 0. For any image to be registered among the N images to be registered, the electronic device transforms the position of the first pixel in the image to be registered according to the position mapping relationship of the first pixel from the reference image to the image to be registered, and obtains a first registered image corresponding to the image to be registered, wherein the first pixel is a pixel matching the reference image and the image to be registered. Then, for the first registered image corresponding to any image to be registered among the N images to be registered, the electronic device adjusts the reference weight of the registered object whose difference value meets the first condition in the first registered image according to the difference value between the registered object of the first registered image and the registered object of the reference image, and obtains a second registered image corresponding to the image to be registered. Compared with the first registered image, the second registered image combines the difference of the registration result, so the negative impact of the unregistered object in the image fusion process can be reduced. Afterwards, the electronic device performs a fusion process on the reference image and the second registered image corresponding to the N images to be registered, to obtain a target image corresponding to the target.

[0006] In this solution, after the above registration step, the N first registered images are not directly fused, but the reference weights of the registered objects whose difference values ​​meet the first condition in the N first registered images are adjusted according to the difference values ​​between the registered objects of the N first registered images and the registered objects of the reference image. Under the influence of the reference weight, the registration objects with different difference values ​​have different degrees of influence on the fusion result. For example, in one implementation, the smaller the reference weight of the registered object (or the registered area) with a larger difference value, the smaller the influence on the fusion result; the larger the reference weight of the registered object with a smaller difference value, the greater the influence on the fusion result. In this way, the importance of different areas to the fusion result in the image process can be adjusted to reduce the negative impact of unregistered objects (or areas) in the image fusion process, thereby improving the problem of image clarity regression after fusion, and making the image processing effect better.

[0007] In another possible design manner of the first aspect, the registration object includes a first pixel point; for the first registered image corresponding to any image to be registered among the N images to be registered, adjusting the reference weight of the registration object whose difference value satisfies the first condition in the first registered image according to the difference value between the registration object of the first registered image and the registration object of the reference image, to obtain the second registered image corresponding to the image to be registered, comprising:

[0008] A first mask mask matrix is ​​constructed, wherein the first mask mask matrix includes a plurality of elements, and the elements in the first mask mask matrix represent the difference values ​​between the first pixel points in the first registered image and the first pixel points in the reference image; according to the difference values, the reference weight of each first pixel point in the first registered image is adjusted to construct a second mask mask matrix, wherein the elements in the second mask mask matrix represent the reference weight of the first pixel points in the first registered image, wherein the reference weight is used to describe the reference degree of the first pixel point in the first registered image when being fused with the first pixel point in the reference image; a second registered image is obtained according to the product value of the first registered image and the second mask mask matrix, and the sum of the product values ​​of the reference image and the third mask mask matrix, wherein the third mask mask matrix is ​​the difference value between the unit matrix of the same order of the second mask mask matrix and the second mask mask matrix.

[0009] In this design, the difference information between the first registered image and the reference image is used to construct a second mask matrix related to the difference information. Each element in the second mask matrix is ​​equivalent to the reference weight of the pixel information of each registered object in the first registered image. Specifically, the first mask matrix is ​​constructed according to the difference value of each registered object in the first registered image, and an element in the first mask matrix represents the difference value of a registered object. The registered object can be a pixel point or a set of pixel points. When the registered object is a set of pixel points, the difference value of the registered object can be the mean, median or maximum value of the sum of the difference values ​​of a set of pixel points.

[0010] After constructing the first mask matrix, taking element a11 of the first mask matrix as an example, the element a11 is replaced with the reference weight a'11 of the registration object corresponding to the element a11, and the other elements of the first mask matrix are processed in the same way to obtain a new matrix, namely, the second mask matrix.

[0011] The second registered image is obtained by first registered image*second mask matrix+reference image*(same order unit matrix-second mask matrix). The difference between the same order unit matrix and the second mask matrix is ​​equivalent to the third mask matrix mentioned above.

[0012] In another possible design manner of the first aspect, adjusting the reference weight of each first pixel in the first registered image according to the difference value includes:

[0013] For any first pixel point in the first registered image, when the difference value between the first pixel point and the second pixel point in the reference image is greater than or equal to a preset threshold, the reference weight of the second pixel point in the first registered image is set to the minimum value in the preset weight value set, wherein the second pixel point is a pixel point that matches any first pixel point in the first registered image.

[0014] Under this design method, if the difference value of a pixel point is greater than a preset threshold, the reference weight of the pixel point is reduced. Therefore, the reference weight of the pixel point is set to the minimum value in the preset weight value set. The preset weight value set can be the y-axis coordinate on the f(x) curve, where f(x) is fitted with (preset threshold, minimum weight) and (difference threshold, maximum weight). For example, when the preset threshold is 10, the minimum weight is 0, the difference threshold is 0, and the maximum weight is 1, f(x) is obtained by fitting the curve based on the two coordinates (10, 0) and (0, 1). The preset weight value set is the y-axis coordinate of the coordinate point on the f(x) curve.

[0015] In another possible design manner of the first aspect, adjusting the reference weight of each first pixel in the first registered image according to the difference value includes:

[0016] For any first pixel point in the first registered image, when the difference value between the first pixel point and the second pixel point of the reference image is less than a preset threshold, the reference weight of the first pixel point of the first registered image is set to a value inversely correlated with the difference value, wherein the second pixel point is a pixel point that matches any first pixel point in the first registered image.

[0017] Under this design, if the difference value is less than the preset threshold, it means that the pixel information of the pixel in the first registered image can be used as a factor affecting the fusion result, and thus the reference weight of these pixels is retained. In order to present the effect that the larger the difference value, the smaller the reference weight; the smaller the difference value, the larger the reference weight, the reference weight of the pixel is set to a value that is inversely correlated with the difference value of the pixel. For example, f(x) mentioned above is an inversely correlated function curve. The difference value of the pixel can be substituted into the x-axis coordinate of the curve, and the obtained y-axis coordinate can be used as the reference weight of the pixel.

[0018] In another possible design manner of the first aspect, the first condition includes: a grayscale difference between a registration object of the first registration image and a registration object of the reference image is greater than 0.

[0019] Among them, the grayscale difference is equal to 0, which means that the first registered image and the registered object are completely consistent with the registered object of the reference image, so no adjustment is made, that is, the pixel value of this type in the first registered image can be directly used for subsequent fusion. This design method processes the reference weights of the registered objects whose grayscale difference is greater than 0, that is, the registered objects that are not fully registered, so that the registered objects that are not fully registered will not interfere with the clarity of the fused image due to excessive difference values.

[0020] In another possible design manner of the first aspect, when the registration object includes at least one first pixel point, the difference value includes: pixel grayscale difference, and the method also includes: obtaining first grayscale information of any first pixel point of the first registration image, and second grayscale information of the second pixel point of the reference image, wherein the second pixel point is a pixel point that matches any first pixel point in the first registration image; performing filtering and smoothing processing on the first grayscale information and the second grayscale information, respectively; obtaining the difference between the first grayscale information after filtering and smoothing processing and the second grayscale information after filtering and smoothing processing, and determining the pixel grayscale difference based on the difference.

[0021] In some designs, the difference value includes a feature difference value, wherein the feature includes at least one of a color feature, a texture feature, a shape feature, and a spatial relationship feature.

[0022] Among them, the method of extracting the grayscale difference of pixel points is more effective and can shorten the image processing time.

[0023] In another possible design manner of the first aspect, determining the grayscale difference of the pixel based on the difference value includes:

[0024] The difference is blurred and smoothed to obtain the grayscale difference of the pixel. The blurred and smoothed processing can avoid the abnormal value (or noise value) in the difference from interfering with the subsequent generation of reference weights, thereby improving the accuracy of the calculation results.

[0025] In another possible design of the first aspect, the noise point position of the reference image is different from the noise point position of the N images to be registered. The different noise point positions mean that, in a set of input image sequences, there is no noise in the A pixel area of ​​one image, and there is noise in the A pixel area of ​​other images, then the fused image should be an image with no noise in the A pixel area.

[0026] In a second aspect, the present application provides an electronic device, comprising: a processor, a memory for storing processor executable instructions, and when the processor is configured to execute instructions, the electronic device implements the method described in the first aspect and any possible design thereof.

[0027] In a third aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible design thereof.

[0028] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any possible design thereof.

[0029] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A process schematic diagram of an image fusion method in the related art;

[0031] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0032] Figure 3A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0033] Figure 4 A set of scenario schematic diagrams provided for embodiments of the present application;

[0034] Figure 5A A schematic diagram of the principle of an image processing method provided in an embodiment of the present application;

[0035] Figure 5B A schematic diagram of another image processing method provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of an image registration system provided in an embodiment of the present application;

[0037] Figure 7 A flowchart of a training method for an image registration neural network model provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a process for obtaining inter-frame differences provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of a mask matrix construction process provided in an embodiment of the present application;

[0040] Fig.10 A schematic diagram of a second registration image provided in an embodiment of the present application;

[0041] Fig.11 A schematic diagram of a fusion process provided in an embodiment of the present application;

[0042] Fig.12 A schematic diagram of a reference image and a target image provided in an embodiment of the present application;

[0043] Fig.13 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0044] Fig.14 A flowchart for generating a fusion mask is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0046] With the continuous development of smart terminal devices, the use of electronic devices is becoming more and more popular. Most electronic devices have built-in cameras, and with the enhancement of the image processing capabilities of electronic devices and the development of camera technology, users pay more and more attention to the camera performance of electronic devices, especially the visual effects of images. Therefore, when the captured image has more noise, blur, and reflection, it will affect the user's shooting experience. Based on this, the feature information of multiple images for the same shooting scene can be fused into one image, so that the fused image contains more information. Ideally, if there is no noise in the A pixel area of ​​one image in a set of input image sequences, and there is noise in the A pixel area of ​​other images (that is, the positions of the noise between the images are different), then the fused image should be an image without noise in the A pixel area. Compared with a noisy image taken after the user presses the shooting button, a better quality image can be taken for the user after multi-image fusion. However, in actual application, the image output by the multi-image fusion technology may not only fail to eliminate noise, but also have lower clarity than the images in the image sequence, and have a serious smearing feeling.

[0047] Figure 1 This is a process diagram of an image fusion method in the related art, refer to Figure 1 , taking the first image 101 as the reference image and the second image 102 as the image to be registered, the first image 101 and the second image 102 are registered, and the first image 101 and the third image 103 are output. One area of ​​the third image 103 is not registered, so Figure 1 It can be seen that the pixel information (such as grayscale value) of the area in the first image 101 and the pixel information in the second image 102 are quite different. If the first image 101 and the third image 103 are directly fused without any processing, the area in the fourth image 104 where the first pixel value 105 should be displayed is covered by the second pixel value 106, resulting in a serious smearing effect on the presented image quality, that is, the image clarity is degraded. Since the image clarity degradation greatly affects the user's photo-taking experience, this problem needs to be solved urgently.

[0048] Among them, poor image registration accuracy is an important reason for the above problems. Image registration is to match and superimpose multiple images acquired at different times, different imaging devices or different shooting conditions (weather, illumination, camera position and angle, etc.). Common methods include optical flow information registration method and grayscale information registration method. The optical flow information registration method uses the change of pixel intensity in two images to calculate the movement of pixels, and then aligns the pixels between the two images to achieve image registration. The grayscale information registration method uses the similarity of pixel grayscale information in two images to determine the mapping of pixels between the two images to achieve image registration. However, when the displacement of some pixels between the two images is too large, or the brightness is inconsistent, the pixel alignment is inaccurate, resulting in a loss of image quality and a smear-like effect on the picture.

[0049] In order to solve the problem of post-fusion image clarity regression caused by poor image registration accuracy, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the electronic device obtains multiple images corresponding to a target object, wherein the multiple images include a reference image and N images to be registered, and N is an integer greater than 0. The image to be registered is used to use the pixel position of the reference image as a reference, and convert the image to be registered into a first configuration image of the same coordinate system as the reference image by transforming the position of the pixel points of the image to be registered. For any image to be registered among the N images to be registered, the electronic device transforms the position of the first pixel point in the image to be registered according to the position mapping relationship of the first pixel point from the reference image to the image to be registered, and obtains a first registered image corresponding to the image to be registered, wherein the first pixel point is a pixel point that matches the reference image and the image to be registered. Then, the electronic device adjusts the reference weight of the registration object whose difference value meets the first condition in the first registration image according to the difference value between the registration object of the first registration image and the registration object of the reference image, for the first registration image corresponding to any image to be registered among the N images to be registered, to obtain the second registration image corresponding to the image to be registered.

[0050] After obtaining the first registered image, this solution processes the first registered image based on the difference between the registered object of the first registered image and the registered object of the reference image to obtain a second registered image, and uses the second registered image as the input for image fusion. Compared with the first registered image, the second registered image combines the difference of the registration results, so it can reduce the negative impact of the unregistered object in the image fusion process. After that, the electronic device fuses the reference image and the second registered image corresponding to the N images to be registered to obtain the target image corresponding to the target.

[0051] In the embodiments of the present application, the electronic device may be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) device, a virtual reality (VR) device, a car computer, or other device with a camera function. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0052] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, referring to Figure 2 Taking the electronic device as a mobile phone as an example, the mobile phone may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0053] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0054] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0055] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0056] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0057] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0058] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0059] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0060] The electronic device implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0061] The external memory interface 220 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0062] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0063] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0064] The electronic device can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0065] The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0066] The camera 293 is used to capture static images or videos. In the preview interface, the camera can capture images. The electronic device can use the images captured within a period of time before and after the user executes the photo control command as the image to be registered or the reference image. After processing, the fused target image is output and displayed on the interface. In some embodiments, the electronic device may include 1 or N cameras 293, where N is a positive integer greater than 1. In some embodiments, the electronic device can use the image captured by one of the cameras 293 as a reference image and the images captured by the other cameras 293 as images to be registered.

[0067] The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV and other formats.

[0068] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0069] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0070] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image registration, image fusion, face recognition, speech recognition, text understanding, etc.

[0071] In some embodiments, a large number of core components in electronic devices, such as processor 210 (central processing unit, image processor, etc.), memory, and various types of host controllers, etc., can be integrated into the main chip. The high integration of the main chip helps to reduce the physical space of the electronic device and meet the current user's demand for lightweight and portable electronic devices.

[0072] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. TM Taking the system as an example, the software structure of the electronic device is illustrated.

[0073] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application, referring to Figure 3The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Android TM The system is divided into four layers, from top to bottom: application layer, application framework layer (referred to as framework layer), Android runtime (Android TM runtime) and system libraries, as well as the kernel layer.

[0074] Among them, the application package can include camera, calendar, map, video, music, short message, gallery, call, desktop, video editing and other applications.

[0075] The framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The framework layer includes some predefined functions. The framework layer may include window manager, content manager, view system, phone manager, resource manager, notification manager, activity manager, etc.

[0076] The view system includes visual controls, such as controls for displaying text and controls for displaying pictures. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying pictures.

[0077] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0078] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0079] The activity manager is used to manage the life cycle of each application and the navigation back function. TM The main thread is created and the life cycle of each application is maintained.

[0080] The kernel layer is the layer between hardware and software. The kernel layer contains at least camera drivers, audio drivers, sensor drivers, etc. The framework layer can call the hardware drivers in the kernel layer, such as calling the camera driver to start the camera in the hardware layer, and the started camera can collect image data.

[0081] In the image processing method provided in the embodiment of the present application, it is detected that the user opens the camera application, the camera application of the application layer starts the photo taking function, and sends instructions to the kernel layer to call the camera driver, sensor driver and display driver of the kernel layer, so that the electronic device starts the camera to collect images. In the process of camera collecting images, light is transmitted to the image sensor through the camera, and the image sensor performs photoelectric conversion on the light signal and converts it into an image visible to the user's naked eye. The output image data is transmitted to the image sensor in the form of a data stream. Figure 3 In the system library, the 3D graphics processing library and the image processing library implement drawing, image rendering, synthesis and layer processing to generate a display layer. The surface manager performs fusion processing on the display layer and passes it to the content provider, window manager and view system of the application framework layer to control the display of the display interface. Finally, the preview image is displayed in the image preview area of ​​the camera application or on the display screen of the electronic device. When the user's trigger operation on the shooting button is detected, the framework layer sends an instruction to the hardware abstract layer (HAL) to call the image processing algorithm, where the HAL layer is a layer located between the system library and the kernel layer. Figure 3 The image processing algorithm in the HAL layer can output a fused target image based on the input reference image and N images to be registered, and display the target image in the shooting interface of the camera application.

[0082] The following takes the electronic device being a mobile phone as an example and describes the method provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0083] An embodiment of the present application provides an image processing method, which is applied to a shooting scene or a video editing scene. Figure 4 A set of scenario diagrams provided for embodiments of the present application, refer to Figure 4, the mobile phone displays the desktop, the camera application 401 is displayed on the desktop, the user's trigger operation on the camera application 401 is detected, the camera application is started, and the preview interface 402 is entered. The preview image collected by the camera is displayed on the preview interface. When the user's trigger operation on the shooting button 403 in the preview interface 402 is detected, the mobile phone obtains a group of image frames 404, wherein different image frames 404 have different noise points 405 due to different shooting angles, shooting times or shooting devices. The mobile phone fuses a group of image frames 404 to obtain a target image 406, and displays a shooting interface 407, and displays the target image 406 on the shooting interface 407. Among them, the specific fusion step includes: taking one of the images in a group of image frames 404 as a reference image, and the other images as images to be registered, taking the reference image as a reference, registering the image to be registered with the reference image to obtain a first registered image set, and adjusting the reference weight when the first registered image is fused according to the difference value between each first registered image and the reference image in the first registered image set, to obtain a corresponding second registered image. The second registered image and the reference image are fused to obtain a target image. Compared with the image quality in the image frame 404, the target image 406 has less noise and the clarity is not degraded, so the picture quality is higher and the shooting effect is better.

[0084] Continue to refer Figure 4 , when a user trigger operation on the video editing application 411 is detected, the video editing application 411 is started and enters the video editing interface. In the video editing interface, the user selects a recorded video 412 and performs denoising on a video frame 413 in the video 412. The mobile phone obtains a group of image frames 414 from the video frame 413, performs fusion denoising on the image frame 414, and obtains the target video frame 415. The mobile phone will replace the selected video frame 413 in the original video 412 with the target video frame 415. Among them, the specific fusion denoising process can refer to the steps of image processing above, and no cumbersome explanation is given here.

[0085] Figure 5A A schematic diagram of the principle of an image processing method provided in an embodiment of the present application, referring to Figure 5A, the method includes four steps, namely, the first image registration, the second image registration, image fusion and image post-processing. Among them, the first image registration is equivalent to the above-mentioned acquisition of multiple frames of images corresponding to the target object, and the multiple frames of images include a reference image and N images to be registered. For any image to be registered among the N images to be registered, the electronic device transforms the position of the first pixel in the image to be registered according to the position mapping relationship of the first pixel from the reference image to the image to be registered, and obtains the first registered image corresponding to the image to be registered. After that, the second image registration is performed according to the difference value, which is equivalent to the above-mentioned acquisition of the difference value between the registered object of the first registered image and the registered object of the reference image, and the reference weight of the registered object whose difference value in the first registered image meets the first condition is adjusted to obtain the second registered image corresponding to the image to be registered. Image fusion refers to the fusion processing of the reference image and the second registered images corresponding to the N images to be registered to obtain the target image corresponding to the target. After the image fusion processing, the image post-processing step can also be performed. In some embodiments, this step can be omitted.

[0086] The embodiment of the present application can achieve the following effects through the second image registration step: even if the registration accuracy is poor, the fused target image can have no loss in quality compared to the original image, and the clarity will not be reduced. These five steps are described in turn below.

[0087] Image Registration:

[0088] In the embodiment of the present application, an image registration system may be used to register multiple images. Figure 6 A schematic diagram of an image registration system provided in an embodiment of the present application, referring to Figure 6 The image registration system includes an image acquisition device 61, a training server 62, and a model application device 63. The image acquisition device 61 acquires multiple images and sends the multiple images to the training server 62. The training server 62 has a built-in neural network model. The training server 62 trains the neural network model with multiple images to obtain a trained image registration neural network model. Afterwards, the model application device 63 can use the image registration neural network model to perform image registration on the reference image acquired by the image acquisition device 61 and the N images to be registered, and output N first registered images.

[0089] The image acquisition device 61, the training server 62 and the model application device 63 may be three separate devices, or may be the same device, or may be the same device in pairs.

[0090] Specifically, taking the image acquisition device 61 and the model application device 63 as the same device as an example, the image acquisition device 61 and the model application device 63 are devices for realizing wireless communication functions, such as terminals, chips used in terminals. Terminals are also called user equipment (UE), electronic devices, etc. Taking the terminal as a mobile phone as an example, the mobile phone can collect multiple images as a sample set, and send the sample set to the training server 62 to train the neural network model to obtain an image registration neural network model. After that, the training server 62 can send the image registration neural network model to the mobile phone, and the mobile phone can, when the reference image and N images to be registered are collected, input the reference image and the N images to be registered into the image registration neural network model for matching processing to obtain the reference image and N first registered images.

[0091] Figure 7 A flowchart of a training method for an image registration neural network model provided in an embodiment of the present application is provided. Figure 7 The training method of the image registration neural network model includes: the image acquisition device sends the collected multiple sets of sample data sets to the training server, wherein each set of sample data sets includes a reference sample image and multiple sample images to be registered. The training server inputs each set of sample data sets into the neural network model, and the neural network model includes a feature extraction network and a regression network. The feature extraction network extracts the pixel information of the reference sample image and the multiple sample images to be registered. The pixel information can be feature information or grayscale information. The feature extraction network inputs the pixel information into the regression network, and the regression network outputs a matching matrix. The matching matrix is ​​used to determine the position mapping relationship of the same pixel information from the reference sample image to the sample image to be registered. According to the matching matrix and the sample image to be registered, the corresponding first registered sample image is obtained. Then, the model parameters are continuously adjusted according to the difference value between the first registered sample image and the reference sample image until the difference value meets the preset conditions, thereby obtaining a trained image registration neural network model.

[0092] In the embodiment of the present application, the neural network model can be constructed based on basic network models such as multilayer perceptron (MLP), convolutional neural network (CNN) and recurrent neural network (RNN). For a specific explanation of the neural network model, please refer to the introduction of related technologies, and this application will not elaborate on it here.

[0093] After the training server trains the image registration neural network model, the model application device (such as a mobile phone) obtains the image registration neural network model from the training server. The image acquisition device acquires a first data set, wherein the first data set includes multiple images. The image acquisition device sends the first data set to the model application device, and the model application device uses one of the images as a reference image and the other images as images to be registered. The model application device uses the image registration neural network model to register the image to be registered and the reference image in the first data set, and outputs a first registered image.

[0094] In some embodiments, the model application device may also send the first registration image and the first data set to the training server to expand the sample set so that the trained image registration neural network model can perform registration processing on images in different scenarios.

[0095] In this embodiment, the reference image may be any image in the first data set. In some embodiments, the reference image may be an image with the highest definition in the first data set. In other embodiments, the reference image may be an image captured by a camera when a user triggers a shooting button. In other embodiments, the reference image may be an image with the least noise in the first data set.

[0096] Exemplarily, the mobile phone can perform clarity detection on the images in the first data set, such as detecting the clarity of the image through edge information and gradient information, and taking the image with the highest or higher clarity as the reference image. It should be noted that the reference image can be one image or multiple images, and this embodiment has no limit on the number of reference images.

[0097] Alternatively, the mobile phone may obtain the current moment when the user triggers the shooting button, and use the image captured at the moment closest to the current moment in the first data set as the reference image, so that the target image finally fused is the image closest to the image captured when the user presses the shooting button.

[0098] Alternatively, the mobile phone may process each image in the first data set in blocks, obtain multiple image blocks corresponding to each image, obtain the variance of pixels in each image block, and use the variance size to represent the noise level of the image block. If the noise level of multiple image blocks in the image is small, the image is used as the reference image.

[0099] After the N first registered images are obtained through registration, the embodiment of the present application does not directly fuse them with the reference image, but performs the following second image registration process, where: Figure 5B As shown, the second image registration process includes a difference extraction step and a mask matrix construction, which can be found in the following description.

[0100] Difference extraction.

[0101] Difference extraction refers to extracting the difference between the registered object in the i-th first registered image and the registered object in the reference image, wherein the difference can be represented by a difference value. i is an integer greater than 0 and less than N.

[0102] The degree of difference is related to the matching accuracy of the image registration neural network model itself. Specifically, the higher the registration accuracy, the smaller the degree of difference, and vice versa. The degree of difference is also related to the shooting scene of the image to be registered and the reference image. For example, the image quality of night scenes and dark light environments is poor, and it is easier to mismatch the positions of pixels, so the difference between the first registered image and the image to be registered is relatively large.

[0103] In some embodiments, when the registration object includes at least one pixel, the difference value includes the grayscale difference of the pixel. Specifically, the mobile phone can obtain the first grayscale information of any first pixel of the first registration image, and the second grayscale information of the second pixel of the reference image, wherein the first pixel is a pixel that matches the image to be registered with the reference image. The second pixel is a pixel that matches any first pixel in the first registration image. Smoothing filtering is performed on the first grayscale information and the second grayscale information respectively. The difference between the first grayscale information after smoothing filtering and the second grayscale information after smoothing filtering is obtained, and the grayscale difference of the pixel is determined based on the difference.

[0104] Figure 8 A schematic diagram of a process for obtaining inter-frame differences provided in an embodiment of the present application, referring to Figure 8 , respectively extract the grayscale information of the reference image, the 1st first registered image, the grayscale information of the n-1th first registered image, and the grayscale information of the nth first registered image. Among them, the grayscale information of the image refers to the grayscale information of the registered object in the image. For example, in this embodiment, the registered object is a pixel point, then the grayscale information of the reference image is the grayscale information corresponding to each pixel point in the reference image, and the grayscale information of the nth first registered image refers to the grayscale information corresponding to each pixel point in the nth first registered image.

[0105] Grayscale information can be obtained by matrix B0~B n The difference between the reference image and the first first registered image is the difference between matrix B0 and matrix B1, which can be represented by matrix A1. The difference between the reference image and the n-1th first registered image is the difference between matrix B0 and matrix B1. n-1 The difference can be expressed by the matrix A n-1 The difference between the reference image and the nth first registered image is represented by the matrix B0 and the matrix B nThe difference can be expressed by the matrix A n Indicates. Among them, the matrix A0~A n Each element in represents the grayscale difference between two pixels that are mapped to each other. The mapping relationship can be obtained from the matching matrix introduced in the above image matching scheme.

[0106] It should be understood that the use of a matrix to store grayscale differences in this embodiment is only an exemplary explanation. In practical applications, key-value pairs, sequences, etc. may also be used for representation, and this application does not limit this. Note: When grayscale differences are represented by key-value pairs, they can be written as [key: value] = [pixel identifier: pixel grayscale difference]. When grayscale differences are represented by sequences, they can be written as [pixel grayscale difference 1, pixel grayscale difference 2…pixel grayscale difference n].

[0107] In some other embodiments, the difference value includes a feature difference value. The feature includes at least one of a color feature, a texture feature, a shape feature, and a spatial relationship feature. The method for extracting the feature difference value is similar to the gray value extraction method described above, and will not be described here in detail.

[0108] In some embodiments, after extracting the pixel information, the pixel information is first denoised and then the difference value is extracted. Specifically, when the pixel information of the pixel value is grayscale information, the grayscale information is denoised; when the pixel information is feature information, the feature information is denoised.

[0109] Among them, the denoising process can be a smoothing filter process. The smoothing filter process can reduce image noise to prevent the noise from affecting the processing result. Smoothing filters include smooth fuzzy mean filter, Gauss filter, median filter, bilateral filter, etc. In practical applications, a suitable smoothing filter operation can be used to denoise the pixel information, and this application does not limit this.

[0110] After the above difference extraction, N groups of pixel difference values ​​can be obtained, and each group of pixel difference values ​​represents the degree of difference between each registered object (or pixel) between a first registered image and a reference image. Afterwards, the embodiment of the present application executes the following step of constructing a mask matrix, thereby assigning different reference weights to the first registered images with different degrees of registration difference, so that the unregistered areas in the first registered image have little effect on the fusion, and the registered areas in the first registered image have a large effect on the fusion, thereby ensuring that when the registration accuracy of the first registered image is low or the clarity of the first registered image is low, the clarity of the fused target image will not regress. The specific construction of the mask matrix can be referred to the introduction below.

[0111] Construct the mask matrix.

[0112] The mask can be understood as being used to block the first registered image (in whole or in part) to control the processing area of ​​the first registered image. For example, when area A in the first registered image is blocked by the mask, area A in the first registered image does not participate in the fusion.

[0113] The mask matrix is ​​a matrix used to indicate whether each pixel in the first registered image needs to be blocked. By multiplying the mask matrix with the first registered image, a second registered image can be obtained by adjusting the first registered image using the difference value as a reference weight. The difference between the second registered image and the first registered image is that there are blocked pixels in the second registered image, while there are no blocked pixels in the first registered image.

[0114] By way of example and not limitation, some specific methods for constructing mask matrices are shown below.

[0115] Example 1:

[0116] The matrices A1 to A in the above text n As the original matrix (or the first mask matrix). n The elements whose difference values ​​are greater than or equal to the preset threshold are set as the minimum value in the preset weights, and the difference value 0 is set as the maximum value in the preset weights. The functional relationship between the reference weight f(x) and the difference value x is fitted, and the reference weights corresponding to other difference values ​​are calculated using this functional relationship. Finally, the reference weight of each pixel value is normalized to obtain the matrix C1~C n (or called the second mask matrix). For example, if the minimum value of the preset weight is set to 0 and the maximum value is set to 1, then A1~A n The elements whose difference value is greater than or equal to the preset threshold are set to 0, and the elements whose difference value is 0 are set to 1; if the minimum value of the preset weight is set to 0.01 and the maximum value is set to 0.99, then A0~A n Elements with difference values ​​greater than or equal to the preset threshold are set to 0.01, and elements with difference values ​​of 0 are set to 0.99.

[0117] In some embodiments, the preset threshold is a natural number less than 255, such as the preset threshold is 5, 10, 15, 20, etc.

[0118] The following takes the preset threshold value as 10 and the minimum value of the preset weight as 0 as an example to illustrate the construction of the mask matrix.

[0119] Fig. 9 A schematic diagram of a mask matrix construction process provided in an embodiment of the present application, refer to Fig. 9In matrix A1, the difference values ​​of the pixels in row 2, column 1, row 2, column 2, and row 4, column 2 are greater than or equal to the preset threshold, so the reference weights of these pixels are set to 0; the difference value of the pixel in row 3, column 3 is 0, so the reference weight of this pixel is set to 1, and the f(x) curve is constructed with x=0, f(x)=1; x=10, f(x)=0. Substitute the difference values ​​of other pixels as x into the f(x) curve to obtain the reference weights of these pixels. After that, the matrix is ​​normalized to obtain C1.

[0120] Example 2:

[0121] Based on Example 1, only matrix A1~A n For the elements whose difference values ​​are greater than or equal to the preset threshold, the reference weights of the remaining elements can be set to a preset value.

[0122] Example 3:

[0123] Based on Example 1, retain the matrices A1~A n For the elements whose difference values ​​are greater than or equal to the preset threshold, the reference weight of the pixel corresponding to the largest difference value is set to 0, and the reference weight of the pixel corresponding to the smallest difference value is set to 1, and the f(x) curve is constructed. The reference weights of the remaining pixels are the vertical axis coordinate values ​​on the curve.

[0124] It should be noted that before constructing the f(x) curve in Example 1 and Example 3, each pixel can be smoothed based on the difference value of all pixels between the i-th first registered image and the reference image, so that the subsequently fitted f(x) curve is smoother and the impact of noise on the fitting curve is reduced.

[0125] In the second mask matrix constructed in this embodiment, the pixels with large difference values ​​in the first registered image have little influence on the fusion result, while the pixels with small difference values ​​in the first registered image have a large influence on the fusion result, thereby improving the problem of clarity regression of the fused target image caused by poor registration accuracy, and achieving better image processing effect.

[0126] Specifically, after obtaining the i-th second mask matrix, substitute it into the following formula to obtain the i-th second registered image.

[0127] The i-th first registered image*the second mask matrix+the reference image*(the same-order unit matrix-the second mask matrix)=the i-th second registered image.

[0128] Compared with the registration accuracy of the i-th first registered image, when the pixels of the i-th second registered image are not registered, the pixel information of the pixels of the reference image is used, and when they are registered, the pixel information of the pixels of the i-th first registered image is used, thereby having a higher registration accuracy.

[0129] Fig.10 A schematic diagram of a second registration image provided in an embodiment of the present application, referring to Fig.10 When the pixel area 1001 of the first registered image is not registered, the pixel information of the pixel area 1001 is replaced with the pixel information of the pixel area 1002, so that the registration accuracy of the pixel area in the second registered image is higher.

[0130] Image fusion.

[0131] Image fusion refers to fusing multiple second registered images and a reference image into a target image. In the embodiment of the present application, a fusion network model can be used for fusion processing.

[0132] The following is an introduction to the training method of the fusion network model. First, multiple sample sets are constructed, each of which includes multiple sample images to be fused and a target sample image. Multiple sample sets are input into the network model for training, and finally a fusion network model is obtained.

[0133] In some embodiments, the fused network model can be trained in a training server. After the training server obtains the fused network model, the model application device (such as a mobile phone) obtains the fused network model from the training server. The model application device inputs the reference image obtained above, the N second registered images and their corresponding N second mask matrices into the fused network model, uses the N second mask matrices as a fusion reference for the fused network model, fuses the reference image and the N second registered images, and obtains the target image.

[0134] Fig.11 A schematic diagram of a fusion process provided in an embodiment of the present application, refer to Fig.11 , N second registered images and the second mask matrix corresponding to each second registered image are obtained from the reference image and N first registered images. The reference image, N second registered images and N second mask matrices are input into the fusion network model, and the target image is output. The pixel information in the target image fuses the pixel information of the reference image and the N images to be registered, so there are fewer noise points, and the target image does not fuse the unregistered objects, so it will not affect the clarity of the image. Compared with the reference image or any image to be registered, the visual effect of the target image is better.

[0135] Fig.12A schematic diagram of a reference image and a target image provided in an embodiment of the present application, referring to Fig.12 , the reference image 1201 contains more noise points, and the visual effect of the image is good. The reference image and the N first registered images are directly fused. Although the noise points are reduced, the pixel area 1203 of the first target image 1202 has a smearing feeling, and the clarity of the pixel area 1201 is lower than that of the reference image. The second target image 1204 obtained by fusing the reference image and N second registered images using the method provided in the embodiment of the present application not only reduces the noise points, but also does not cause clarity regression (or image quality loss). Therefore, the embodiment of the present application can adjust the importance of different areas to the fusion result in the image process by reference weights to reduce the negative impact of the unregistered areas in the image fusion process, thereby improving the problem of clarity regression of the fused image and making the image processing effect better.

[0136] In some embodiments, pixel weighted average image fusion may be used, that is, pixel values ​​of the reference image and pixel values ​​of corresponding pixels of the first registered image of the i-th frame are weighted averaged to obtain new pixel values. Alternatively, a transform domain-based image fusion method may be used. For a specific explanation of the fusion method, reference may be made to the introduction of related technologies, and this application will not elaborate on it here.

[0137] After this step, the fused target image can be output to the mobile phone display interface. Alternatively, before sending it to the display, the mobile phone also performs the following image post-processing operation to obtain a target image with higher picture quality.

[0138] Image post-processing.

[0139] In some embodiments, the mobile phone may perform denoising on the target image and output the denoised image. For details, please refer to the description related to the smoothing filter processing above.

[0140] Of course, image post-processing may also include processing the brightness and size of the image, adding filters to the image, etc., and this application does not impose any restrictions on this.

[0141] For example, please refer to Fig.13, which is a process diagram of an image processing method provided by an embodiment of the present application. In this method, a first image set (equivalent to the multiple images mentioned above) is first obtained, wherein the first data set includes N+1 frames of input images, and the first image set is a plurality of images acquired after detecting the trigger operation of the user on the shooting button. The first frame of input image is used as the reference image, and the second frame of input image, the third frame of input image...the N+1 frame of input image are used as the first frame of image to be registered, the second frame of image to be registered...the Nth frame of image to be registered. Based on the reference image, the N images to be registered are processed by a registration algorithm (equivalent to the image registration neural network model introduced above), and the reference image and N first registered images are registered. The difference values ​​between the first first registered image and the reference image, the difference values ​​between the second first registered image and the reference image...the difference values ​​between the Nth first registered image and the reference image are extracted respectively. N fusion masks (equivalent to the second mask matrix introduced above) are constructed based on the difference values. According to the i-th first registered image * i-th fusion mask + reference image * (1-i-th fusion mask), the i-th second registered image is obtained. The N second registered images are concat-processed and input into the fusion network model together with the N fusion masks for image fusion to obtain the first target image. Optionally, the first target image is denoised to obtain the second target image.

[0142] exist Fig.13 In the method shown in FIG. 1 , the difference between the reference image and the i-th first registration image and the i-th fusion mask can be extracted by the following method: Fig.14 , which is a flow chart for generating a fusion mask provided by an embodiment of the present application. In this method, the grayscale information of the reference image and the grayscale information of the i-th first registered image are calculated respectively, and the grayscale information is smoothed and filtered respectively. Then, the grayscale difference is obtained, wherein the grayscale difference refers to the grayscale difference between two pixels (or the first pixel points in the above text) that are mapped to each other. The grayscale difference greater than or equal to the preset threshold is threshold-cut by a preset threshold, wherein the threshold truncation refers to the elimination of the grayscale difference greater than or equal to the preset threshold. Then, blur smoothing (or mean smoothing) is performed on the remaining grayscale difference. Then, the grayscale value is normalized to generate a 0-1 fusion mask. This step can refer to the construction of the mask matrix described above. Then, the 0-1 fusion mask is used to replace the area with low registration accuracy (i.e., large grayscale difference) in the i-th first registered image with the corresponding area of ​​the reference image to obtain a second registered image. After the N second registered images are merged and concat-processed, they are input into the fusion network model together with N 0-1 fusion masks to obtain the first target image.

[0143] In summary, the method provided in the embodiment of the present application does not directly input the registered N first registered images and the reference image into the fusion network after the image to be registered has been registered, but uses the difference information between the N first registered images and the reference image to construct a second mask matrix related to the difference information, and each element in the second mask matrix is ​​equivalent to the reference weight of the pixel information of each pixel in the first registered image. Therefore, the second mask matrix is ​​sent into the fusion network model as auxiliary information, which can enhance the pixel information of the image fused with the reference image (equivalent to the second registered image above), reduce the negative impact of the unregistered area in the image fusion process, thereby improving the problem of image clarity regression after fusion, and making the image processing effect better.

[0144] Some other embodiments of the present application provide an electronic device, which may include: a display screen (such as a touch screen), a memory, and one or more processors. The display screen, the memory, and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above method embodiment. The structure of the electronic device can refer to Figure 4 The structure of the electronic device shown.

[0145] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0146] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0147] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0152] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire multiple images corresponding to the target object, wherein the multiple images include a reference image and N images to be registered, where N is an integer greater than 0; For any image to be registered among the N images to be registered, according to a position mapping relationship of the first pixel from the reference image to the image to be registered, the position of the first pixel in the image to be registered is transformed to obtain a first registered image corresponding to the image to be registered, wherein the first pixel is a pixel that matches the reference image and the image to be registered; For a first registered image corresponding to any image to be registered among the N images to be registered, according to the difference value between the registered object of the first registered image and the registered object of the reference image, the reference weight of the registered object whose difference value satisfies the first condition in the first registered image is adjusted to obtain a second registered image corresponding to the image to be registered, wherein the registered object includes a first pixel point, and obtaining the second registered image corresponding to the image to be registered includes: constructing a first mask matrix, the first mask matrix including a plurality of elements, the elements in the first mask matrix representing the difference value between the first pixel point in the first registered image and the first pixel point in the reference image; according to the difference value, Adjust the reference weight of each first pixel in the first registered image to construct a second mask matrix, wherein the elements in the second mask matrix represent the reference weights of the first pixels in the first registered image, wherein the reference weights are used to describe the reference degree of the first pixels in the first registered image when being fused with the first pixels in the reference image; obtain a second registered image according to the product value of the first registered image and the second mask matrix, and the sum of the product values ​​of the reference image and a third mask matrix, wherein the third mask matrix is ​​the difference between the unit matrix of the same order of the second mask matrix and the second mask matrix; The reference image and the second registered images corresponding to the N images to be registered are fused to obtain a target image corresponding to the target.

2. The method according to claim 1, characterized in that The step of adjusting the reference weight of each first pixel in the first registered image according to the difference value includes: For any first pixel point in the first registered image, when the difference value between the first pixel point and the second pixel point in the reference image is greater than or equal to a preset threshold, the reference weight of the second pixel point in the first registered image is set to the minimum value in the preset weight value set, wherein the second pixel point is a pixel point that matches any first pixel point in the first registered image.

3. The method according to claim 2, characterized in that The step of adjusting the reference weight of each first pixel in the first registered image according to the difference value includes: For any first pixel point in the first registered image, when the difference value between the first pixel point and the second pixel point of the reference image is less than a preset threshold, the reference weight of the first pixel point of the first registered image is set to a value inversely correlated with the difference value, wherein the second pixel point is a pixel point that matches any first pixel point in the first registered image.

4. The method according to any one of claims 1 to 3, characterized in that The first condition includes: a grayscale difference between a registration object of the first registration image and a registration object of the reference image is greater than 0.

5. The method according to any one of claims 1 to 4, characterized in that In the case where the registration object includes at least one first pixel point, the difference value includes: a pixel grayscale difference, and the method further includes: Acquire first grayscale information of any first pixel of the first registered image and second grayscale information of a second pixel of the reference image, wherein the second pixel is a pixel that matches any first pixel in the first registered image; performing smoothing filtering processing on the first grayscale information and the second grayscale information respectively; The difference between the first grayscale information after the smoothing filter processing and the second grayscale information after the smoothing filter processing is obtained, and the grayscale difference of the pixel is determined based on the difference.

6. The method according to claim 5, characterized in that The determining the pixel grayscale difference based on the difference value includes: The difference is subjected to blur smoothing processing to obtain the pixel grayscale difference.

7. The method according to any one of claims 1 to 6, characterized in that The position of the noise points of the reference image is different from the positions of the noise points of the N images to be registered.

8. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor, wherein when the processor is configured to execute the instructions, the electronic device implements the method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on the electronic device, cause the electronic device to execute the method according to any one of claims 1 to 7.

Images