An image processing method and apparatus

Abstract

The application provides an image processing method and device, relates to the technical field of multimedia, and is used for fast distortion correction processing of an image, reduction of calculation time consumption, and improvement of the real-time performance of image distortion correction processing. The method comprises the following steps: performing grid encoding on an initial image to obtain an encoding grid corresponding to different regions of the initial image; performing indexing on the encoding grid of the initial image in an image encoding library to obtain a library grid displacement vector field corresponding to a library encoding grid with the highest matching degree; obtaining a full-image grid displacement vector field corresponding to the initial image according to the encoding grid of the initial image and the library grid displacement vector field; and performing mapping processing on each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after correction processing.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Based on the imaging principle of cameras, the larger the field of view of a camera, the more objects it captures, and the more information it can hold in a single photograph. Therefore, more and more electronic devices are equipped with wide-angle or ultra-wide-angle cameras to achieve shooting scenarios with a large field of view, such as group photos. However, the captured images generally suffer from wide-angle distortion, and the larger the camera's field of view, the more severe the wide-angle distortion, especially in areas far from the center. For example, Figure 1 In the image shown, the face is elongated and distorted, and the window frame in the background is warped.

[0003] Current classic distortion correction algorithms primarily obtain optical distortion and wide-angle distortion coefficients through parameters such as the camera's field of view, and then map them to distortion correction coefficients for image coordinate transformation to perform full-image distortion correction. However, the resulting image performs poorly in terms of local shape preservation and field of view maintenance.

[0004] Another wide-angle portrait distortion correction scheme is based on region detection. This scheme constructs distortion-preserving terms for both the head and background regions, along with field-of-view distortion-preserving terms and a smoothing constraint term for the entire image. Different weights are assigned to the different distortion-preserving terms, and joint optimization is performed to obtain the displacement vector field of the entire image. Finally, the original image is mapped using this displacement vector field to obtain the corrected target image. However, this optimization algorithm is complex and computationally time-consuming, making real-time processing difficult. It cannot display the distortion-corrected image in real-time on the preview interface when the user is taking photos or videos. Summary of the Invention

[0005] This application provides an image processing method and apparatus that reduces computation time and improves the real-time performance of image distortion correction processing through a fast algorithm for distortion correction processing.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, an image processing method is provided, which includes: performing grid encoding on an initial image to obtain an encoding grid corresponding to different regions in the initial image; indexing the encoding grid of the initial image in an image encoding library to obtain a library grid displacement vector field corresponding to a library encoding grid with the highest matching degree; obtaining a full-image grid displacement vector field corresponding to the initial image according to the encoding grid of the initial image and the library grid displacement vector field; and performing mapping processing on each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after rectification processing.

[0008] In the above technical solution, the initial image is grid encoded with a relatively coarse granularity, the library grid displacement vector obtained by indexing the encoding grid in the image encoding library is used as the initial displacement vector, and then the full-image grid displacement vector field corresponding to the initial image is calculated according to the library grid displacement vector field, so that the mapping processing of the pixel points of the initial image is performed to realize the distortion correction or deformation processing of the image. This implementation has a relatively low calculation complexity, can quickly process the preview image or the image sequence in the video recording, and improves the real-time performance of the image distortion rectification processing.

[0009] In a possible implementation, the image encoding library includes a plurality of library encoding grids and a library grid displacement vector field corresponding to each library encoding grid.

[0010] In the above possible implementation, the preconfigured image encoding library can be used to index the input initial image to obtain the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree as the initial grid displacement vector field of the initial image, so that the algorithm complexity can be optimized and the real-time performance of the image processing can be improved.

[0011] In a possible implementation, the indexing of the encoding grid of the initial image in the image encoding library to obtain the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree specifically includes: determining, as the library encoding grid with the highest matching degree with the encoding grid of the initial image, the library encoding grid with the minimum first difference function between the encoding grid of the initial image and the library encoding grid; and indexing the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree in the image encoding library.

[0012] In the above possible implementation, the matching indexing is performed according to the difference between the encoding grid of the initial image and the library encoding grid, and the library encoding grid with the minimum difference function is used as the library encoding grid with the highest matching degree, so that the library grid displacement vector field closest to the input image is obtained, the algorithm performance is good, and the real-time performance of the image processing is improved.

[0013] In a possible implementation, the full-image grid displacement vector field corresponding to the initial image is obtained according to the library grid displacement vector field, and specifically includes: adjusting and calculating the library grid displacement vector field according to a second difference function between the encoding grid of the initial image and the library encoding grid, to obtain the full-image grid displacement vector field corresponding to the initial image.

[0014] In the possible implementation, considering that there is a certain difference between the matched library grid displacement vector field and the input initial image, the displacement vector field can be finely adjusted according to the difference function between the encoding grid of the initial image and the library encoding grid and the difference function between the displacement vectors, to obtain a more accurate full-image displacement vector field corresponding to the initial image, thereby optimizing and improving the accuracy and rationality of image processing and improving the user experience.

[0015] In a possible implementation, the adjusting and calculating of the library grid displacement vector field according to the second difference function between the encoding grid of the initial image and the library encoding grid specifically includes: dividing the target encoding grid corresponding to the feature region of the initial image into three types of grid points, wherein the grid points of the target encoding grid that overlap with the library encoding grid are type 1 grid points, the grid points included in the library encoding grid and not included in the target encoding grid are type 2 grid points, and the grid points included in the target encoding grid and not included in the library encoding grid are type 3 grid points; calculating the displacement vectors of the type 1 grid points in the initial image, the type 3 grid points in the initial image, and the grid points corresponding to the feature points in the initial image; interpolating and updating the displacement vectors of the type 3 grid points in the initial image according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image encoding library; interpolating to obtain the displacement vectors of the type 2 grid points in the image encoding library according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image encoding library and the difference between the displacement vectors of the type 3 grid points in the initial image and the image encoding library; interpolating to obtain the displacement vectors of the grid points in the background region of the initial image according to the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image encoding library, and the type 3 grid points in the initial image, and the displacement vectors of the feature points in the initial image; and synthesizing the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image encoding library, and the type 3 grid points in the initial image, the displacement vectors of the feature points in the initial image, and the displacement vectors of the grid points in the background region, to obtain the full-image displacement vector field.

[0016] In the possible implementation manner, the fine adjustment method for the grid displacement vector of the input image based on the difference between the input image grid and the library grid enables a displacement vector field matched with the input image grid to be solved even when the input image grid and the library grid are not completely consistent. Thus, the final optimized image has a good wide-angle distortion correction effect on a local interested part, a smooth transition between a background area and a correction area, and consistency between front and back frames of the image sequence. The simplified calculation algorithm avoids a large calculation amount of the full-image grid displacement vector field, and effectively improves the real-time performance of the image distortion correction processing.

[0017] In a possible implementation manner, after obtaining the full-image grid displacement vector field corresponding to the initial image according to the library grid displacement vector field, the method further includes: performing tracking correction processing on the full-image displacement vector field to obtain a corrected full-image grid displacement vector field; and performing mapping processing on each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after correction processing, specifically including: performing mapping processing on each pixel point of the initial image according to the corrected full-image grid displacement vector field to obtain the target image after correction processing.

[0018] In the possible implementation manner, the continuous transition relationship of the displacement vector fields of the front and back frames of the image sequence is utilized, and the tracking correction processing is performed on the full-image displacement vector fields corresponding to the front and back frames of the image sequence, so that the consistency of the fast processing of the image sequence can be further maintained, and the continuity and consistency of the distortion correction or deformation processing of the video image can be improved, and the user's shooting experience can be improved.

[0019] In a second aspect, an image processing apparatus is provided, which includes: a preprocessing module configured to grid encode an initial image to obtain encoded grids corresponding to different regions in the initial image; a library indexing module configured to index the encoded grids of the initial image in an image encoding library to obtain a library grid displacement vector field corresponding to a library encoded grid with the highest matching degree; a displacement calculation module configured to obtain a full-image grid displacement vector field corresponding to the initial image according to the encoded grids of the initial image and the library grid displacement vector field; and a mapping processing module configured to perform mapping processing on each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after correction processing.

[0020] In a possible implementation manner, the image encoding library includes a plurality of library encoded grids and library grid displacement vector fields corresponding to the library encoded grids.

[0021] In a possible implementation, the library index module is specifically configured to: determine a library coding grid with the minimum first difference function between the coding grid of the initial image and the library coding grid as the library coding grid with the highest matching degree to the coding grid of the initial image; and index the library grid displacement vector field corresponding to the library coding grid with the highest matching degree in the image coding library.

[0022] In a possible implementation, the displacement calculation module is specifically configured to: calculate the adjustment of the library grid displacement vector field according to a second difference function between the coding grid of the initial image and the library coding grid, to obtain the full-image grid displacement vector field corresponding to the initial image.

[0023] In a possible implementation, the displacement calculation module is specifically configured to: divide the target coding grid corresponding to the feature region of the initial image into three types of grid points, wherein the grid points overlapped by the target coding grid and the library coding grid are type 1 grid points, the grid points included in the library coding grid and not included in the target coding grid are type 2 grid points, and the grid points included in the target coding grid and not included in the library coding grid are type 3 grid points; calculate the displacement vectors of the type 1 grid points in the initial image, the type 3 grid points in the initial image, and the grid points corresponding to the feature points in the initial image; calculate and update the displacement vectors of the type 3 grid points in the initial image by interpolation according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image coding library; calculate the displacement vectors of the type 2 grid points in the image coding library by interpolation according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image coding library, and the difference between the displacement vectors of the type 3 grid points in the initial image and the image coding library; calculate the displacement vectors of the grid points in the background region of the initial image by interpolation according to the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image coding library, and the type 3 grid points in the initial image, and the displacement vectors of the feature points in the initial image; and obtain the full-image displacement vector field by synthesizing the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image coding library, the type 3 grid points in the initial image, the displacement vectors of the feature points in the initial image, and the displacement vectors of the grid points in the background region.

[0024] In a possible implementation, the device further includes a tracking correction module configured to perform tracking correction processing on the full-image displacement vector field to obtain a corrected full-image grid displacement vector field; and the mapping processing module is specifically configured to: perform mapping processing on each pixel point of the initial image according to the corrected full-image grid displacement vector field to obtain a target image after correction processing.

[0025] In a third aspect, an electronic device is provided, which includes: a processor; a memory for storing instructions executable by the processor; and wherein the processor is configured to execute the instructions to implement the method according to any one of the first aspect.

[0026] In a fourth aspect, a computer-readable storage medium is provided, characterized by instructions in the computer-readable storage medium, when executed by a processor of an electronic device, cause the electronic device to perform the method of any one of the above first aspect.

[0027] In a fifth aspect, a computer program product is provided, characterized by, when the computer program product is run on a computer, cause the computer to perform the method of any one of the above first aspect.

[0028] For the understanding, any one of the above provided image processing apparatus, computer-readable storage medium or computer program product can be realized by the corresponding method provided above, therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A hardware system architecture diagram of an electronic device provided by an embodiment of the present application;

[0030] Figure 2 A software system architecture diagram of an electronic device provided by an embodiment of the present application;

[0031] Figure 3 A schematic diagram of an image processing application scenario provided by an embodiment of the present application;

[0032] Figure 4 A flowchart of an image processing method provided by an embodiment of the present application;

[0033] Figure 5 A schematic diagram of grid processing of an initial image in an image processing method provided by an embodiment of the present application;

[0034] Figure 6 A matching schematic diagram of a target coding grid and a library coding grid in an image processing method provided by an embodiment of the present application;

[0035] Figure 7 A calculation schematic diagram of a full image displacement vector field in an image processing method provided by an embodiment of the present application;

[0036] Figure 8 A flowchart of another image processing method provided by an embodiment of the present application;

[0037] Figure 9 A structural schematic diagram of another image processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The terms "first", "second", etc. are used only for the purpose of description and should not be interpreted in a relative way unless otherwise specified or two or more is explicitly indicated. Thus, features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0039] It should be noted that in this application, the words "exemplary" or "for example" are used to mean "an example of" or "an example, only. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be interpreted as more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0041] The image processing method provided by the embodiments of the present application can be applied to electronic devices with shooting and image processing capabilities. The electronic devices can be mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, vehicle-mounted devices, ultra-mobile personal computers (UMPC), netbooks, cellular phones, personal digital assistants (PDA), augmented reality (AR) \ virtual reality (VR) devices, smart televisions, smart cameras, monitoring devices, etc. The specific form of the electronic device is not specially limited in the embodiments of the present application.

[0042] Figure 1 A structural schematic diagram of the electronic device 100 is shown.

[0043] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0045] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0046] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0047] The processor 110 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can call them directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0048] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0049] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the photographing function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0050] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0051] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0052] The wireless communication function of the electronic device 100 can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0053] The mobile communication module 150 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 100. The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0054] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light emitting diode (AMOLED), flex light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 can include 1 or N display screens 194, and N is a positive integer greater than 1.

[0055] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0056] ISP is used to process the data feedback from the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and conversion into a visible image. ISP can also optimize the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, ISP can be provided in the camera 193.

[0057] The camera 193 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or other format image signal. In some embodiments, the electronic device 100 can include one or N cameras 193, such as a long-focus camera, a medium-focus camera, a short-focus (wide-angle) camera, or a wide-angle black-and-white camera, etc., where N is a positive integer greater than 1.

[0058] In some embodiments of the present application, the electronic device 100 can include a wide-angle camera or a super-wide-angle camera. The electronic device can generate a real-time preview image based on the image captured by the wide-angle camera. Further, the electronic device can also perform real-time wide-angle distortion correction processing, or personalized deformation and correction processing, etc., based on the image obtained by at least one camera, and update the target image after processing as the preview image in real time, thereby improving and enriching the user's shooting experience.

[0059] NPU is a neural-network (NN) computing processor that processes input information quickly by borrowing from the structure of biological neural networks, such as the transmission mode between human brain neurons, and can also continuously self-learn. Through NPU, the electronic device 100 can realize intelligent cognition and other applications, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0060] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (such as a sound playing function, an image playing function, etc.) required by a function, etc. The data storage area can store data (such as audio data, a phone book, etc.) created during use of the electronic device 100, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0061] In the embodiments of the present application, the internal memory 121 described above can store computer program codes for implementing the steps in the method embodiments of the present application. The processor 110 described above can run the computer program codes of the steps in the method embodiments of the present application stored in the memory 121. The display screen 194 described above can be used to display a shooting object of a camera, a real-time video frame, etc. involved in the embodiments of the present application.

[0062] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0063] Figure 2 is a software structure block diagram of the electronic device 100 in the embodiments of the present application.

[0064] The layered architecture divides software into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Android runtime and a system library, and a kernel layer.

[0065] The application layer can include a series of application packages. As shown in Figure 2 the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. The embodiments of the present application are mainly implemented by improving the camera application in the application layer, for example, by adding a plug-in to the camera to expand its functions.

[0066] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0067] As shown in Figure 2 The application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like. In embodiments of the present application, the program of the camera of the application layer can be improved through the application framework layer, so that the target object can display a special effect image or a special effect video of the motion trail of the target object in the display screen 194 during shooting. The special effect image or the special effect video is synthesized by the background of the electronic device through real-time calculation and processing.

[0068] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, and the like.

[0069] The content provider is used to store and obtain data, and make the data accessible to the application. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, and the like.

[0070] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, and the like. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0071] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including call connection, call hang-up, and the like).

[0072] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

[0073] The notification manager enables the application to display notification information in the status bar. The notification manager can be used to convey a notification type of message, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloading, message reminders, and the like. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application running in the background, and can also be a notification in the form of a dialogue window appearing on the screen. For example, the text information is prompted in the status bar, a prompt sound is emitted, the electronic device is vibrated, the indicator light flashes, and the like.

[0074] The Android Runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0075] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0076] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0077] The system library can include multiple functional modules. For example: surface manager, media library, three-dimensional graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL) and the like.

[0078] The surface manager is used to manage the display subsystem, and provides a fusion of 2D and 3D layers for multiple applications.

[0079] The media library supports multiple commonly used audio, video format playback and recording, and static image files and the like. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG and the like.

[0080] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing and the like.

[0081] The 2D graphics engine is a drawing engine for 2D drawing.

[0082] The kernel layer is the layer between hardware and software, and can also be referred to as the driver layer. The kernel layer at least contains display drivers, camera drivers, audio drivers, and sensor drivers.

[0083] The working flow of the software and hardware of the electronic device 100 is exemplarily explained below in combination with a capture and photographing scene.

[0084] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, a timestamp of the touch operation, and the like). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, and identifies a control corresponding to the input event. Taking an example in which the touch operation is a touch click operation, and the control corresponding to the click operation is a control of a camera application icon, the camera application invokes an interface of the application framework layer, starts the camera application, and then starts a camera driver by invoking the kernel layer, and captures a still image or a video by the camera 193.

[0085] In the embodiment of the present application, when a still image or a video is captured by the camera 193 during the process in which the user uses the electronic device to capture a photo or a video, the captured image or video can be temporarily stored in the content provider. When a capturing operation is performed, the photo or the video captured can be displayed by the view system. In the embodiment of the present application, the image needs to be processed by full-image fast deformation correction of multiple images before being displayed in the preview interface.

[0086] The technical solutions involved in the following embodiments can be implemented in the electronic device 100 with the hardware architecture and the software architecture described above. The following embodiments of the present application will be described in detail in combination with the drawings, taking an example in which the electronic device 100 is a mobile phone.

[0087] For example, the user can instruct the mobile phone to start the camera application by touching a specific control on the screen of the mobile phone, pressing a specific physical button or a combination of buttons, inputting a voice instruction, a gesture in the air, and the like. In response to the instruction of the user to start the camera application, the mobile phone starts the camera and displays a capturing interface.

[0088] For example, the user can instruct the mobile phone to start the camera application by clicking the "camera" application icon on the desktop of the mobile phone, and the mobile phone displays the capturing interface. Alternatively, when the mobile phone is in a locked state, the user can instruct the mobile phone to start the camera application by swiping rightward on the screen of the mobile phone, or by swiping upward, and the mobile phone displays the capturing interface. For another example, when the mobile phone is in the locked state, the user can instruct the mobile phone to start the camera application by clicking the shortcut icon of the "camera" application on the locked interface, and the mobile phone can also display the capturing interface. For yet another example, when the mobile phone is running other applications, the user can instruct the mobile phone to start the camera application to capture by clicking a control of the camera function.

[0089] The photographing interface of the electronic device generally includes a real-time image preview frame, a photographing control, and other function controls, for example, can include "large aperture", "portrait", "photograph" or "video recording", etc. The real-time image preview frame can be used to preview the real-time image collected by the camera, and the user can determine the timing of instructing the phone to perform a photographing operation based on the image in the real-time image preview frame, so as to generate a target image. The user instructing the phone to perform a photographing operation can be, for example, the user clicking the photographing control, or the user pressing the volume key, etc.

[0090] For example, taking a mobile phone equipped with a short-focus (wide-angle) camera, a medium-focus camera, and a long-focus camera as an example. In the case where the relative position of the mobile phone and the photographed object does not change, the short-focus (wide-angle) camera has the smallest focal length, the largest field of view (FoV), and the smallest object size in the photographed image. The medium-focus camera has a larger focal length than the short-focus (wide-angle) camera, a smaller field of view than the short-focus (wide-angle) camera, and a larger object size in the photographed image than the short-focus (wide-angle) camera. The long-focus camera has the largest focal length, the smallest field of view, and the largest object size in the photographed image. For different framing areas, the mobile phone can intelligently select different camera combinations to ensure imaging quality and composition effect.

[0091] The field of view refers to, in an optical instrument, the angle formed by the two edges of the largest range of the measured target image that can pass through the lens of the optical instrument with the lens as the vertex. The field of view can be used to indicate the maximum angle range that can be photographed by the camera during the process of photographing an image. That is, if the object to be photographed is within this angle range, the object to be photographed will be captured by the mobile phone, and if the object to be photographed is not within this angle range, the object to be photographed will not be captured by the mobile phone. Generally, the larger the field of view of the camera, the larger the range of the camera. The smaller the field of view of the camera, the smaller the range of the camera.

[0092] That is, the size of the field of view determines the field of view of the optical instrument, the larger the field of view, the larger the field of view, and the smaller the optical magnification. The wide-angle camera generally refers to a camera with a field of view of 60 degrees or more, which can capture a larger range of images, but the image at close range may be distorted. The main camera in the electronic device can be a wide-angle lens with a focal length of 35 mm or less. In addition, the electronic device can also be configured with an ultra-wide-angle lens with a focal length of 24 mm or less, etc.

[0093] In the embodiments of this application, during the process of a user taking photos or videos using an electronic device, a real-time fast image processing method can be used to perform wide-angle distortion correction processing on the image generated by the wide-angle camera, and display the corrected image in real time on the shooting preview interface, thereby improving the user's shooting experience and enhancing the image quality output by the wide-angle camera. Furthermore, the fast image processing method provided in this application can also be used to perform real-time personalized deformation processing on the image acquired by the camera, such as real-time face slimming, body shaping, head replacement, or body part replacement, enriching the user's shooting experience.

[0094] The embodiments of this application can be applied to, but are not limited to, the following two image processing scenarios:

[0095] Scenario 1: When electronic devices take photos or record videos, preview the target image after distortion correction or deformation processing in real time.

[0096] For example, when a user takes a photo using the front or rear camera of their phone, in wide-angle shooting mode, they can enable wide-angle distortion correction by clicking the "Real-time Correction" icon, or the phone can automatically enable wide-angle distortion correction upon detection. The phone reads the preview video stream and performs wide-angle distortion correction on the head and body areas in the image in real time. The processed image is then displayed on the shooting preview interface; the image presented on the user interface at this point has already undergone distortion correction. Afterward, when the user clicks the photo or video recording button, the photo or video saved on the phone is the target image or video obtained after image correction.

[0097] Scenario 2: A scenario in which distortion correction or deformation processing is achieved by post-processing existing images or videos.

[0098] For example, users can use the "Gallery" or "Photos" app on their electronic devices to select images or videos for distortion correction or deformation processing.

[0099] like Figure 3 As shown, the electronic device enters the following state based on the image selected by the user: Figure 3 The "Smart Correction" interface shown contains user-adjustable correction parameters. Additionally, it may include textures and other materials for deformation processing.

[0100] For example, the "Portrait Distortion Adjustment" parameter bar can be used to adjust the degree of overall portrait distortion correction, the "Head Distortion Adjustment" parameter bar can be used to adjust the degree of head distortion correction, and the "Body Distortion Adjustment" parameter bar can be used to adjust the degree of body distortion correction. The electronic device recommends a suitable degree of correction by default, and users can manually adjust any of these parameters to change the distortion correction in two directions: positive and negative.

[0101] When the user adjusts the parameters through the parameter bar on the above interface, the electronic device performs correction or deformation processing on the imported initial image according to the adjusted parameters, and then updates the output target image and displays it in the image frame interface. The user can save the processing effect of the current interface by clicking the "save" button.

[0102] In addition, in the above scenarios one and two, personalized deformation processing can also be included, for example, the user can replace a funny "dog head" sticker on a real-time photographed portrait or a stored portrait, or replace a body part area of the portrait with a funny action sticker, and the like. In this case, the user can select an image area for deformation processing through a funny sticker gallery or plug-in of the interface, to achieve funny deformation processing of the image.

[0103] It should be noted that the above scenario examples and the interface designs shown in the above figures are only used as examples to introduce the embodiments of the present application, and the interface of the electronic device can be any interface design containing the above functions. In addition, the head and body in the above scenarios can be replaced with other target objects to achieve similar correction or deformation effects. The present application does not make specific limitations on the above interface designs and target subjects of image processing. Figure 3

[0104] On the basis of the above hardware and software involved in the embodiments of the present application, in combination with the above shooting scenarios, the embodiments of the present application will be described in detail in combination with the corresponding drawings. As shown in the above figures, the method can include: Figure 4

[0105] 401: The electronic device extracts a feature region of an initial image to obtain a feature point set.

[0106] The initial image can be a static image collected by a camera configured by the electronic device, or a stored image of the electronic device. In addition, since a video file includes a sequence of images frame by frame, the initial image can be one frame of a video file collected by a camera configured by the electronic device, or one frame of a stored video file of the electronic device.

[0107] The electronic device can identify the background image region and the foreground image region in the initial image according to a preconfigured algorithm, so as to extract a region of interest in the foreground image region, i.e., the feature region of the initial image, to obtain a feature point set corresponding to the target feature.

[0108] ​​For example, if the initial image is a portrait, the electronic device can extract the image region other than the portrait from the initial image as the background region, and extract the target feature regions such as the head region and body region from the portrait as the foreground region. At the same time, the electronic device can extract the feature points of facial features such as eyes, nose, and mouth from the head region, and the limb contours or postures from the body region, and store the feature point set corresponding to the target features of the initial image.

[0109] 402: The electronic device performs gridding processing on the initial image and obtains the corresponding coded grid.

[0110] In the embodiments of this application, in order to reduce the computational load of image processing and improve the real-time performance of image correction processing, the electronic device can perform gridding and grid coding on the initial image. That is, different regions of the initial image are mapped to multiple grid codes, and the matching calculation for correction processing of the initial image is performed with grid points of larger granularity, thereby reducing the computational load of image processing in subsequent steps. The granularity of the coding grid is larger than that of the image's pixel coding.

[0111] Among them, gridded coding is a method of dividing the image coding region into grids of equal size according to the latitude and longitude, and then performing multi-value coding on the coding grids of different feature regions.

[0112] like Figure 5 As shown, the electronic device divides the initial image into grids and can perform multi-value encoding on the foreground region (including the head region, body region, etc.) and the background region according to the extracted image regions and grid encoding rules. For example, the grid of the background region can be encoded as 0, the grid of the head region can be encoded as 1, and the grid of the body region can be encoded as 2. In this example, the grid points with non-zero grid encoding values ​​(grid encoding as 1 and grid encoding as 2) can be referred to as the target encoded grid.

[0113] In one implementation, the coding grid corresponding to the feature regions in the initial image can be referred to as the target coding grid. For example, in the example above, the grid with non-zero coding values ​​is the target coding grid.

[0114] This gridded encoding method encodes the original value of each grid point from 0 to 255 into a smaller value, such as 0 to 3, according to different regions. Multiple grid points in the same region correspond to one encoded value, thus reducing the complexity of representing the entire image. Therefore, the gridded encoding of the initial image in this application to obtain the encoded grid (or grid points) can also be called sparse encoding or sparse representation of the image.

[0115] 403: The electronic device indexes the initial image according to the encoding grid to obtain a library grid displacement vector field corresponding to the highest matching degree in the image encoding library.

[0116] The image encoding library is a database storing a plurality of sample images, (library) encoding grids, (library) grid displacement vector fields, and corresponding relationships between the (library) encoding grids and the (library) grid displacement vector fields, and is preconfigured on the electronic device. The (library) encoding grids and the (library) grid displacement vector fields have a one-to-one correspondence.

[0117] The grid displacement vector field is used to indicate the planar distribution parameters of the displacement vectors of the grid points on the image, and is used to represent a set of displacement vectors of the grid points required for the correction processing of the image. Since the encoding grid and the grid displacement vector field are one-to-one, the grid displacement vector field corresponding to an encoding grid includes displacement vectors of a plurality of grid points, and the displacement vectors corresponding to each grid point are substantially different. In addition, the displacement vectors included in the grid displacement vector field are non-discrete values. Relative to the sparse representation of the image corresponding to the above-mentioned grid encoding, the displacement vector field can be referred to as a dense representation of the image in the present application.

[0118] Specifically, the electronic device can perform grid encoding on a large number of sample images, and perform corresponding distortion correction or deformation processing, thereby obtaining the encoding grids and the corresponding grid displacement vector fields of a large number of sample images, establishing an image encoding library, and thereby being able to match the encoding grid of the input initial image with the corresponding library grid displacement vector field in the image encoding library.

[0119] For example, the image encoding library can store the encoding grids corresponding to the avatars of a plurality of face shapes and expressions, and the grid displacement vector fields corresponding to the encoding grids. The image encoding library can also include the encoding grids and the corresponding grid displacement vector fields of the portraits of a plurality of postures and body postures, and can also include the encoding grids and the corresponding grid displacement vector fields of images of other scenes, or the encoding grids and the corresponding grid displacement vector fields of a plurality of interesting stickers, so as to meet the needs of the user for image correction or deformation processing.

[0120] In an embodiment, considering the space occupation problem of the electronic device storing the image encoding library, and the limitation of the calculation amount of the index processing of the initial image encoding grid based on a large image encoding library, the preconfigured image encoding library on the electronic device can be a subset of the global image encoding library obtained according to the above-mentioned embodiment, that is, a subset selected by a certain algorithm and capable of representing the global image encoding library. For example, the electronic device can obtain a subset with low repetition through a maximum correlation algorithm or a minimum redundancy algorithm as the preconfigured image encoding library.

[0121] Based on this, the electronic device can query one or more library coding grids with the highest matching degree from the image coding library according to the coding grid of the input initial image, and index the library grid displacement vector field corresponding to the one or more library coding grids.

[0122] In an implementation, the electronic device can perform the matching query according to the closest library coding grid under certain distance constraints. Alternatively, the electronic device can obtain the library coding grid with the highest matching degree based on the first difference function between the coding grid of the initial image and the library coding grid being the smallest. Then the electronic device can index the library grid displacement vector field corresponding to the library coding grid with the highest matching degree in the image coding library. The specific matching algorithm can refer to related technologies, and the present application does not make specific limitations thereto.

[0123] For example, as shown in FIG. 1, Figure 6 Figure 6 The left image in FIG. 1 schematically shows the grid point with a coding grid of 1 of the initial image, Figure 6 The right image in FIG. 1 schematically shows the grid point of the library coding grid with the highest matching degree to the grid point with a coding grid of 1 of the initial image in the image coding library.

[0124] In addition, the electronic device can update the image coding library according to the obtained initial image and the image after distortion correction, so that the samples in the image coding library are closer to or match the image processing habits or use scenarios of the user.

[0125] 404: The electronic device obtains the full-image grid displacement vector field corresponding to the initial image according to the coding grid of the initial image and the library grid displacement vector.

[0126] Further, the electronic device can adjust and calculate the library grid displacement vector field according to the second difference function between the coding grid and the library coding grid, to obtain the full-image grid displacement vector field corresponding to the initial image.

[0127] That is, the electronic device matches the grid displacement vector field (dense expression) in the image coding library through the coding grid (sparse expression) of the initial image, and then finely adjusts the matched grid displacement vector field (dense expression) according to the difference between the coding grid (sparse expression) of the initial image and the library coding grid (library sparse expression) in the image coding library, to adapt to the problem that the sparse expression of the initial image and the library sparse expression are not completely consistent, so that the initial image can be distortion corrected or image deformed according to the full-image grid displacement vector field obtained after the fine adjustment.

[0128] ​The second difference function can be a difference function between the initial image coding grid and the library coding grid, that is, the second difference function can be the same as the first difference function in the step 403. In addition, the second difference function can also be different from the first difference function, for example, the second difference function can include a difference function between the initial image coding grid and the library coding grid and a difference function between the displacement vector between the initial image coding grid and the library coding grid.

[0129] In an embodiment, the electronic device can calculate the full image displacement vector field corresponding to the initial image according to the similarity between the initial image coding grid and the library coding grid. That is, the electronic device can calculate the displacement vector field of the grid points of the initial image based on the library coding grid corresponding to the image coding library matched in the step 403 and the grid points overlapped and the grid points not overlapped between the library coding grid and the feature region of the initial image.

[0130] In summary, first, the electronic device can match the library coding grid closest to the target coding grid of the input initial image from the image coding library; then, calculate the displacement vector field of the grid coding value non-0 and the feature points in the initial image based on the classical distortion mapping relationship; divide the grid points included in the target coding grid in the initial image into three types: the shared grid points of the target coding grid and the library coding grid, the unique grid points of the target coding grid, and the unique grid points of the library coding grid; based on the difference between the shared grid points of the target coding grid and the library coding grid and the displacement vector of the feature points, interpolate to calculate the displacement vector of the unique grid points of the target coding grid; based on the difference between the calculated displacement vector between the target coding grid and the library coding grid and the difference between the displacement vector of the feature points, interpolate to calculate the displacement vector of the unique grid points of the library coding grid; based on the union grid of the calculated grid points of the target coding grid and the library coding grid and the feature points, combine the displacement vector of the boundary grid, and interpolate to calculate the displacement vector field of the initial image coding grid; and finally obtain the displacement vector field of the full image of the input initial image.

[0131] The calculation process of the electronic device to obtain the full image grid displacement vector field corresponding to the initial image according to the initial grid point displacement vector (i.e., the library grid displacement vector) of the initial image will be described in detail below.

[0132] In an embodiment, the electronic device can divide each target corresponding to the foreground region of the initial image into three types based on the grid points corresponding to the target coding grid of the initial image. Among them, type 1 is the grid point of the overlapping region of the grid point corresponding to the target coding grid of the initial image and the grid point corresponding to the library coding grid, type 2 is the grid point unique to the library coding grid, and type 3 is the grid point unique to the target coding grid of the initial image. Then, the electronic device can calculate the full image displacement vector field according to the similarity of the regions covered by the grid points between the target coding grid of the initial image and the library coding grid.

[0133] For example, based on the above Figure 5 The target coding grid of the initial image shown in the figure is that the background region is coded as 0, the avatar region is coded as 1, and the body region is coded as 2. Combined with Figure 6 As shown, the electronic device obtains the library coding grid matched in the image coding library according to the initial image.

[0134] As shown in Figure 6 The electronic device can divide the grid points of the initial image with grid coding 1 into the above three types. Among them, the grid points of type 1 represent the grid points that overlap each other in the image coding library and the grid of the initial image. The grid points of type 1 in the initial image can be represented as GI shared , and the grid points of type 1 in the image coding library can be represented as GD shared . The grid points of type 2 represent the grid points unique to the image coding library, that is, the input initial image does not include the grid points of type 2. The grid points of type 2 in the image coding library can be represented as GD only . The grid points of type 3 represent the grid points unique to the input initial image, that is, the image coding library does not include the grid points of type 3. The grid points of type 3 in the image coding library can be represented as GI only .

[0135] Then, the electronic device can calculate the full image displacement vector field according to the similarity of the regions covered by the grid points of type 1, type 2 and type 3. For example, as shown in Figure 7 The specific calculation process can be realized by the following algorithm.

[0136] 1. Calculate the displacement vectors of the grid points corresponding to type 1 in the initial image and type 3 in the initial image, and the feature points.

[0137] Specifically, the displacement vectors of the grid points GI shared of type 1 and the grid points GI only of type 3 in the initial image, and the feature points FI are calculated, and the calculated displacement vectors can be represented as GIV shared , GIV onlyand FIV. Wherein, the coordinates of the feature points are not necessarily on the grid points.

[0138] It should be noted that the displacement vector corresponding to the grid point can be calculated according to a classical distortion mapping algorithm, such as a spherical projection or a Mercator projection algorithm, and the specific algorithm and calculation process are not limited in the present application.

[0139] For example, the displacement vector calculated by the electronic device for the type 1 grid point, the type 3 grid point and the feature point in the kth region of the initial image using the classical wide-angle distortion mapping method can be represented as:

[0140] GIV shared (k)=f1 k (GI shered (k)),

[0141] GIV only (k)=f1 k (GI only (k)),

[0142] FIV(k)=f1 k (FI(k))。

[0143] Wherein, f1 k (G) represents a function of the distortion mapping algorithm used by the electronic device for the grid point G of the kth region. For different k, the f1 k (G) functions can be the same or different, to adapt to the electronic device which can set different processing rules or processing degrees for different image regions.

[0144] 2. According to the difference between the displacement vector of the type 1 grid point of the initial image and the displacement vector of the grid point in the image encoding library, the displacement vector of the type 3 grid point in the initial image is calculated and updated by using an interpolation method.

[0145] In addition, the electronic device can also calculate the displacement vector of the type 1 grid point in the initial image and the displacement of the feature point in the initial image by using the difference.

[0146] Wherein, the interpolation method can use a classical interpolation mapping method based on feature points, such as thin-plate spline transformation, or radial basis function mapping algorithm.

[0147] For example, the electronic device calculates the displacement vector GIV shared (k) of the type 1 grid point in the initial image according to the difference between the displacement vector GIV shared (k) of the corresponding grid point in the image encoding library, and the displacement vector GIV2 only(k), and the displacement vector GIV2 of the grid point of type 1 in the initial image shared (k), and the displacement vector FIV2 of the feature point. Specifically, it can be expressed as:

[0148] GIV2 only (k) = f2 k (fd1 k (GIV shared (k), GDV shared (k)), GIV only (k)),

[0149] GIV2 shared (k) = f2 k (fd1 k (GIV shared (k), GDV shared (k)), GIV shared (k)),

[0150] FIV2(k) = f2 k (fd1 k (GIV shared (k), GDV shared (k)), FIV(k)).

[0151] wherein f2 k (G) represents a function of the interpolation mapping method adopted for the grid point G of the kth region, and the interpolation mapping algorithm can not be limited to the thin-plate spline transformation algorithm. fd1 k (G1, G2) represents a function of calculating the difference between the grid point G1 and the grid point G2 of the kth region, and the difference calculation function can include but not limited to the numerical difference of the displacement amount in the X-axis and Y-axis directions. For different k, f2 k (G) or fd1 k (G1, G2) can be the same or different, so as to adapt to the different processing rules or processing degrees that the electronic device can set for different image regions.

[0152] 3. According to the difference between the displacement vector of the grid point of type 1 in the initial image and the image encoding library, and the difference between the displacement vector of the grid point of type 3 in the initial image and the image encoding library, the displacement vector of the grid point of type 2 in the image encoding library is calculated by using the interpolation method.

[0153] Specifically, the electronic device can interpolate the displacement vector of the grid point of type 2 in the image encoding library according to the difference between the displacement vector of the grid point of type 1 in the image encoding library and the displacement vector of the grid point of type 1 in the initial image calculated in the foregoing step 2, and the difference between the displacement vector of the grid point of type 3 in the image encoding library and the displacement vector of the grid point of type 1 in the initial image calculated in the foregoing step 2.

[0154] For example, the electronic device can interpolate the displacement vector GDV2(k) of the grid point of type 2 in the image encoding library according to the difference between the displacement vector GDV1(k) of the grid point of type 1 in the image encoding library and the displacement vector GIV2(k) of the grid point of type 1 in the initial image calculated in the foregoing step 2, and the difference between the displacement vector GDV3(k) of the grid point of type 3 in the image encoding library and the displacement vector GIV2(k) of the grid point of type 3 in the initial image. shared shared only only only

[0155] only k k shared shared only only only

[0156] k k k k

[0157] 4. Interpolating the displacement vector of the grid point of the background region of the initial image according to the displacement vector of the grid point of type 1 in the initial image, the displacement vector of the grid point of type 2 in the image encoding library, the displacement vector of the grid point of type 3 in the initial image, the displacement vector of the feature point in the initial image, and the displacement vector of the boundary grid point.​​​​​​​​​​​​​​​​​​​

[0158] The initial value of the displacement vector of the grid point in the background region of the initial image (for example, the grid code is 0 in the foregoing example) can be 0. The initial value of the displacement vector of the boundary grid point in the initial image can also be 0, or the initial value of the displacement vector of the boundary grid point can be calculated by the method of moving on the boundary contour of the initial image.

[0159] For example, the electronic device can calculate the displacement vector GIV2 shared (k) of the type 1 grid point in each image region of the initial image only (k) and the displacement vector GIV2 only (k) of the type 3 grid point, and combine the displacement vector GBV border of the boundary grid point to calculate the displacement vector GV2 all of the grid point in the background region (that is, the grid code is 0) of the initial image by interpolation.

[0160]

[0161] The initial value of GV all can be 0. f4(G1, G2,...Gn) represents an interpolation mapping algorithm for integrating all regions for full image interpolation mapping algorithm, which can not be limited to thin-plate spline transformation and the like. w k represents a weight coefficient of the distortion correction degree of each region of the initial image, fd3 k (G1, G2, G3) represents a distance calculation function of the displacement vectors of the grid points G1, G2, and G3 in the kth region. fd3 k (G1, G2, G3) can be the same or different, to adapt to the electronic device that can set different processing rules or processing degrees for different image regions.

[0162] 5. The displacement vector field of the full image is obtained by integrating the displacement vectors of the type 1 grid point in the initial image, the type 2 grid point in the image code library, the type 3 grid point in the initial image, and the displacement vectors of the feature points in the initial image, the displacement vectors of the grid points in the background region.

[0163] Specifically, the electronic device can calculate the displacement vector GIV2 shared (k) of the type 1 grid point, the displacement vector GDV2 only (k) of the type 2 grid point, and the displacement vector GIV2 only (k) of the type 3 grid point, combine the displacement vector GBV border of the boundary grid point, and calculate the displacement vector GV2 all, jointly constitute a full-image displacement vector field. In the calculation of the full-image displacement vector field, the plurality of encoding regions can be calculated as a whole.

[0164] 405: The electronic device maps each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after rectification processing.

[0165] The electronic device can map the initial image according to the calculated full-image grid displacement vector field to obtain a mapping function for processing each pixel point. That is, the electronic device needs to obtain the displacement vector of each pixel point in the initial image based on the displacement vector of the grid point, so as to warp map the full-image pixel point. Wherein, the warp mapping can specifically be changing the coordinates of each pixel point or the pixel value of each pixel point according to the displacement amount of each pixel point around the grid point and the plane position relationship between the pixel point and these surrounding grid points, so as to obtain a target image after distortion correction or deformation processing.

[0166] In addition, the electronic device can also perform remapping processing on the initial image according to the user-selected map based on the above algorithm. Wherein, remapping refers to the process of copying the pixels of a certain position in an image to the specified position in another picture. In this application, the specific operation of remapping is that the electronic device maps each pixel in the user-selected map (selected from the map database) to the user-specified position in the initial image. Thus, the electronic device can perform interesting map processing on the initial image according to the user's operation to enrich the user's shooting experience and improve the shooting interest.

[0167] Through the above image processing method provided by the present application, the electronic device can quickly correct the distortion caused by the wide-angle lens in the preview image or the image sequence in the video recording, so that the interesting area in the real-time preview image after correction processing has no distortion effect, and the user can achieve the effect of what you see is what you get when taking pictures or recording videos, improving the user's subjective visual effect. In addition, the electronic device can also set different distortion correction parameters for different regions to realize customized correction or deformation processing, and adaptively adjust the effect and degree of correction or deformation processing, further enriching the user's experience.

[0168] In addition, compared with the existing distortion correction processing scheme, the application provides a method of grid matching through an image coding library, simplifies the calculation algorithm, avoids the large calculation amount of the full image grid displacement vector field, and effectively improves the real-time performance of the image distortion correction processing. Through the fine adjustment method of the grid displacement vector of the input image based on the difference between the input image grid and the library grid, even if the input image grid and the library grid are not completely consistent, a displacement vector field matched with the input image grid can be solved. Thus, the final optimized image has good wide-angle distortion correction effect in the local interested part, the background area and the correction area are smoothly transitioned, and the correction effect between the front and rear frames of the image sequence is consistent. In addition, in the process of processing the sample image to establish the image coding library, the edge of the sample image can be processed to preserve the edge, so that the image obtained by the above-mentioned embodiment of the application has no loss in the image viewing angle.

[0169] In addition, the application also provides another image processing method. In the case that the electronic device is not configured with an image coding library, or the index of the image coding library does not obtain a library coding grid matched with the initial image, the distortion correction or deformation processing of the image can also be realized.

[0170] In a possible implementation, in the case that the electronic device is not configured with an image coding library, or the index of the image coding library does not obtain a library coding grid matched with the initial image, the electronic device can execute steps 401-402 in the above-mentioned embodiment, and then execute steps 1-5 in the above-mentioned embodiment 404 to obtain the full image grid displacement vector field corresponding to the initial image. Figure 7 In the case that the electronic device is not configured with an image coding library, the initial grid point displacement vector (i.e., the library grid displacement vector) is not obtained through the matching index based on the coding grid of the initial image, and in the algorithm of steps 1-5, the displacement vector of the grid point of the image coding library can be set to 0, so that the full image grid displacement vector field corresponding to the initial image can be obtained according to the above-mentioned algorithm. The specific algorithm is described in detail above, and will not be described here. Finally, the electronic device executes step 405 as described in the above-mentioned embodiment, that is, the electronic device maps each pixel point of the initial image according to the full image grid displacement vector field to obtain the target image after the correction processing.

[0171] Through the above embodiments, the electronic device can, in the case that the image encoding library is not pre-configured or there is no matching result for the index, gradually calculate the grid displacement vector field of the full image based on the encoding grid after grid encoding on the input initial image, so as to realize the distortion correction or deformation processing of the image, and the algorithm process is relatively simplified, and the real-time performance of image processing is improved.

[0172] Based on the above embodiments, in the case that the user shoots a video through the electronic device, there is continuity between the front and rear frames of the image sequence, and the electronic device needs to keep the consistency of the distortion correction or deformation processing of the front and rear frames in consideration of the shooting or viewing experience of the user. Therefore, the electronic device can further constrain the feature points of the front and rear frames of the image, increase the tracking correction of the displacement vector field, and thus better ensure the consistency of the distortion correction or deformation processing of the front and rear frames.

[0173] In an embodiment, after step 404, that is, after the electronic device obtains the full image grid displacement vector field corresponding to the initial image according to the library grid displacement vector field, the method can further include: Figure 8

[0174] 801: The electronic device performs tracking correction processing on the full image displacement vector field to obtain a corrected full image grid displacement vector field.

[0175] Specifically, for the image sequence, after the electronic device obtains the full image displacement vector field corresponding to the current frame according to step 404, the electronic device performs tracking correction processing on the calculated full image displacement vector field.

[0176] The tracking algorithm refers to that the electronic device can predict the full image displacement vector field of the current frame in combination with the full image displacement vector fields corresponding to the previous frames, and correct the full image displacement vector field of the current frame in combination with the calculated full image displacement vector field of the current frame, to obtain a corrected full image grid displacement vector field.

[0177] For example, Kalman filtering algorithm, mean filtering algorithm, or median filtering algorithm, etc.

[0178] 802: The electronic device performs mapping processing on each pixel point of the initial image according to the corrected full image grid displacement vector field to obtain a target image after correction processing.

[0179] ​In the embodiments described above, the electronic device utilizes the continuous transition relationship of the displacement vector fields of the preceding and following frames of an image sequence to perform tracking and correction processing on the full-image displacement vector fields corresponding to the preceding and following frames of the image sequence. This further maintains the consistency of rapid processing of the image sequence, thereby improving the continuity and consistency of distortion correction or deformation processing of video images and enhancing the user's shooting experience.

[0180] This application also provides an image processing apparatus, such as... Figure 9 As shown, the device 900 includes a preprocessing module 901, a library indexing module 902, a displacement calculation module 903, and a mapping processing module 904.

[0181] The preprocessing module 901 can be used to perform gridding encoding on the initial image to obtain the encoded grid corresponding to different regions in the initial image.

[0182] The library index module 902 can be used to index the image coding library based on the coding grid of the initial image to obtain the library grid displacement vector field corresponding to the library coding grid with the highest matching degree.

[0183] The displacement calculation module 903 can be used to obtain the full-image grid displacement vector field corresponding to the initial image based on the encoded grid of the initial image and the library grid displacement vector field.

[0184] The mapping processing module 904 can perform mapping processing on each pixel of the initial image according to the full-image grid displacement vector field to obtain the corrected target image.

[0185] In one possible implementation, the image coding library may include multiple library coding grids and the corresponding library grid displacement vector fields.

[0186] In one possible implementation, the library index module 902 can be specifically used to: determine the library coding grid with the smallest first difference function between the coding grid of the initial image and the library coding grid as the library coding grid with the highest matching degree with the coding grid of the initial image; and index the library grid displacement vector field corresponding to the library coding grid with the highest matching degree in the image coding library.

[0187] In one possible implementation, the displacement calculation module 903 can be specifically used to: adjust and calculate the displacement vector field of the library grid according to the second difference function between the coding grid of the initial image and the library coding grid, so as to obtain the full-image grid displacement vector field corresponding to the initial image.

[0188] In a possible implementation, the displacement calculation module 903 can be specifically configured to: divide the target coding grid corresponding to the feature region of the initial image into three types of grid points, where the grid points of the target coding grid overlapping with the library coding grid are type 1 grid points, the grid points included in the library coding grid and not included in the target coding grid are type 2 grid points, and the grid points included in the target coding grid and not included in the library coding grid are type 3 grid points; calculate the displacement vectors of the type 1 grid points in the initial image, the type 3 grid points in the initial image, and the grid points corresponding to the feature points in the initial image; according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image coding library, the displacement vectors of the type 3 grid points in the initial image are calculated and updated by interpolation; the displacement vectors of the type 2 grid points in the image coding library are calculated by interpolation according to the difference between the displacement vectors of the type 1 grid points in the initial image and the image coding library, and the difference between the displacement vectors of the type 3 grid points in the initial image and the image coding library; the displacement vectors of the background region grid points of the initial image are calculated by interpolation according to the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image coding library, and the type 3 grid points in the initial image, and the displacement vectors of the feature points in the initial image; and the full-image displacement vector field is obtained by synthesizing the displacement vectors of the type 1 grid points in the initial image, the type 2 grid points in the image coding library, the type 3 grid points in the initial image, the displacement vectors of the feature points in the initial image, and the displacement vectors of the background region grid points.

[0189] In a possible implementation, the apparatus 900 can further include a tracking correction module configured to perform tracking correction processing on the full-image displacement vector field to obtain a corrected full-image grid displacement vector field; and the mapping processing module is specifically configured to: perform mapping processing on each pixel point of the initial image according to the corrected full-image grid displacement vector field to obtain a target image after correction processing.

[0190] In a possible implementation, the apparatus 900 can be configured to perform the steps performed by the electronic device in the foregoing embodiments, for example, steps 401 to 405 in the foregoing embodiments.

[0191] In combination with the foregoing Figure 1 As shown in the electronic device schematic diagram, the electronic device can be the image processing apparatus (electronic device) provided in the embodiments of the present application. As shown in Figure 1 The electronic device 100 includes a processor 110 and an internal memory 121. The processor 110 and the internal memory 121 can communicate with each other through an internal connection path to transfer control and / or data signals. The internal memory 121 can be configured to store a computer program, and the processor 110 can be configured to call and run the computer program from the internal memory 121 to perform the steps in the foregoing embodiments to implement image processing.

[0192] In particular, the electronic device 100 can correspond to various embodiments of the method according to the embodiments of the present application. Moreover, the various units in the electronic device 100 and the other operations and / or functions described above are respectively configured to implement the corresponding procedures in various embodiments of the method.

[0193] The processor 110 described above can be configured to perform one or more processing actions implemented by the electronic device described in the foregoing method embodiments. For details, please refer to the description in the foregoing method embodiments, which will not be repeated here.

[0194] It should be understood that the processor in the embodiments of the present application can be a CPU, and the processor can also be other general-purpose processors, digital signal processing (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc.

[0195] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically EPROM (EEPROM) or flash memory. The volatile memory can be random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRAM) (DRAM).

[0196] The embodiment of the present application further provides a computer readable medium for storing computer program codes, the computer program codes comprising instructions for executing the method in the network device and the terminal device.

[0197] The present application further provides a computer program product comprising instructions which, when executed, cause the terminal device and the network device to perform operations corresponding to the terminal device and the network device of the above method.

[0198] Optionally, the computer instructions are stored in a storage unit.

[0199] Optionally, the storage unit is a storage unit configured for the electronic device, such as a register, a cache, etc., and can also be a storage unit outside a chip in the device, such as a ROM or other type of static storage device that can store static information and instructions, a RAM, etc. The processor mentioned in any of the above can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above feedback information transmission method. The processing unit and the storage unit can be decoupled and arranged on different physical devices, and are connected through wired or wireless means to realize the respective functions of the processing unit and the storage unit to support the system chip to realize various functions in the above embodiments. Alternatively, the processing unit and the storage unit can be coupled on the same device. It should be understood that the processor in the embodiments of the present application can be a CPU, and the processor can also be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0200] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described modules, devices and units can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0201] It should be understood that in various embodiments of the present application, the size of the serial number of the above processes does not mean the execution order, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0202] Those skilled in the art can understand that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0203] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical functional division, and actual implementation can have another division manner, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection between some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0204] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical units, that is, they can be located in one place, or can be distributed to multiple modules. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0205] In addition, the functional modules in each embodiment of the present application can be integrated into a processing module, or each module can exist physically, or two or more modules can be integrated into one unit.

[0206] When the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.

[0207] Finally, it should be noted that: the above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: grid coding of an initial image to obtain an encoding grid corresponding to different regions in the initial image; indexing the encoding grid of the initial image in an image encoding library to obtain a library grid displacement vector field corresponding to the library encoding grid with the highest matching degree; wherein the image encoding library comprises a plurality of library encoding grids, and a library grid displacement vector field corresponding to the library encoding grid; obtaining a full-image grid displacement vector field corresponding to the initial image according to the difference between the encoding grid of the initial image and the library grid displacement vector field; mapping processing of each pixel point of the initial image according to the full-image grid displacement vector field to obtain a target image after correction processing.

2. The method of claim 1, wherein, The indexing of the encoding grid of the initial image in the image encoding library to obtain the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree comprises: determining the library encoding grid with the minimum first difference function between the encoding grid of the initial image and the library encoding grid as the library encoding grid with the highest matching degree with the encoding grid of the initial image; indexing in the image encoding library to obtain the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree.

3. The method according to claim 1 or 2, characterized in that, The obtaining of the full-image grid displacement vector field corresponding to the initial image according to the difference between the encoding grid of the initial image and the library grid displacement vector field comprises: adjusting and calculating the library grid displacement vector field according to a second difference function between the encoding grid of the initial image and the library encoding grid to obtain the full-image grid displacement vector field corresponding to the initial image.

4. The method of claim 3, wherein, The adjusting and calculating of the library grid displacement vector field according to the second difference function between the encoding grid of the initial image and the library encoding grid comprises: dividing a target encoding grid corresponding to a feature region of the initial image into three types of grid points, wherein the grid points overlapping the target encoding grid and the library encoding grid are type 1 grid points, the grid points included in the library encoding grid but not included in the target encoding grid are type 2 grid points, and the grid points included in the target encoding grid but not included in the library encoding grid are type 3 grid points; calculating displacement vectors of type 1 grid points in the initial image, type 3 grid points in the initial image, and grid points corresponding to feature points in the initial image; interpolating and updating the displacement vectors of type 3 grid points in the initial image according to the difference between the displacement vectors of type 1 grid points in the initial image and the image encoding library; interpolating to obtain the displacement vectors of type 2 grid points in the image encoding library according to the difference between the displacement vectors of type 1 grid points in the initial image and the image encoding library, and the difference between the displacement vectors of type 3 grid points in the initial image and the image encoding library; interpolating to obtain the displacement vectors of background region grid points according to the displacement vectors of type 1 grid points in the initial image, type 2 grid points in the image encoding library, and type 3 grid points in the initial image, and the displacement vectors of feature points in the initial image; The displacement vector field of the whole image grid is obtained by synthesizing the displacement vectors of the grid points of type 1 in the initial image, the grid points of type 2 in the image encoding library, the grid points of type 3 in the initial image, and the displacement vectors of the feature points in the initial image, and the displacement vectors of the grid points in the background region.

5. The method according to claim 1 or 2, characterized in that, After the whole image grid displacement vector field corresponding to the initial image is obtained according to the difference between the encoding grid of the initial image and the library grid displacement vector field, the method further comprises: performing tracking correction processing on the whole image grid displacement vector field to obtain a corrected whole image grid displacement vector field; then the mapping processing of each pixel point of the initial image according to the whole image grid displacement vector field to obtain a target image after correction processing, specifically comprises: mapping processing of each pixel point of the initial image according to the corrected whole image grid displacement vector field to obtain a target image after correction processing.

6. An image processing apparatus characterized by comprising: The device comprises: a preprocessing module for grid encoding of the initial image to obtain the encoding grid corresponding to different regions in the initial image; a library index module for indexing the encoding grid of the initial image in the image encoding library to obtain the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree; wherein the image encoding library comprises a plurality of library encoding grids, and the library grid displacement vector field corresponding to the library encoding grid; a displacement calculation module for obtaining the whole image grid displacement vector field corresponding to the initial image according to the difference between the encoding grid of the initial image and the library grid displacement vector field; a mapping processing module for mapping processing of each pixel point of the initial image according to the encoding grid of the initial image and the whole image grid displacement vector field to obtain a target image after correction processing.

7. The apparatus of claim 6, wherein, The library index module is specifically configured to: determine the library encoding grid with the minimum first difference function between the encoding grid of the initial image and the library encoding grid as the library encoding grid with the highest matching degree with the encoding grid of the initial image; index the library grid displacement vector field corresponding to the library encoding grid with the highest matching degree in the image encoding library.

8. The apparatus of claim 6 or 7, wherein, The displacement calculation module is specifically configured to: adjust and calculate the library grid displacement vector field according to the second difference function between the encoding grid of the initial image and the library encoding grid to obtain the whole image grid displacement vector field corresponding to the initial image.

9. The apparatus of claim 8, wherein, The displacement calculation module is specifically configured to: divide the target encoding grid corresponding to the feature region of the initial image into three types of grid points, wherein the grid points of type 1 are the grid points overlapping the library encoding grid, the grid points of type 2 are the grid points included in the library encoding grid and not included in the target encoding grid, and the grid points of type 3 are the grid points included in the target encoding grid and not included in the library encoding grid; calculate the displacement vectors of the grid points of type 1 in the initial image, the grid points of type 3 in the initial image, and the grid points corresponding to the feature points in the initial image; According to the difference between the initial image and the displacement vector of the grid point of type 1 in the image coding library, the displacement vector of the grid point of type 3 in the initial image is calculated and updated by interpolation; According to the difference between the initial image and the displacement vector of the grid point of type 1 in the image coding library, and the difference between the initial image and the displacement vector of the grid point of type 3 in the image coding library, the displacement vector of the grid point of type 2 in the image coding library is calculated by interpolation; According to the displacement vectors of the grid points of type 1 in the initial image, type 2 in the image coding library and type 3 in the initial image, and the displacement vector of the feature point in the initial image, the displacement vector of the background grid point of the initial image is calculated by interpolation; The displacement vector field of the full image grid is obtained by synthesizing the displacement vectors of the grid points of type 1 in the initial image, type 2 in the image coding library, type 3 in the initial image, and the displacement vector of the feature point in the initial image, and the displacement vector of the background grid point.

10. The apparatus of claim 6 or 7, wherein, The device further comprises: a tracking correction module for tracking and correcting the full image grid displacement vector field to obtain a corrected full image grid displacement vector field; Then the mapping processing module is specifically used for: According to the corrected full image grid displacement vector field, each pixel point of the initial image is mapped to obtain a target image after correction processing.

11. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the method of any one of claims 1 to 5.

13. A computer program product, characterised in that, When the computer program product is running on the computer, the computer can execute the method of any one of claims 1 to 5.

Images