Image Processing Method and Electronic Device

Abstract

The present application relates to an image processing method and an electronic device. The image processing method includes: acquiring a first image through a camera, where the first image is one frame of a plurality of consecutive frames acquired by the camera; predicting, based on the first image, a pixel offset of a second image after the first image in the plurality of frames relative to the first image, where the pixel offset is a spatial position change of pixels of the second image relative to the first image; obtaining a first enhancement coefficient based on the first image and the pixel offset; acquiring the second image through the camera; correcting the contrast of the second image according to the first enhancement coefficient; and displaying the corrected second image. Correcting the enhancement coefficient according to the spatial position change of the pixels can minimize the spatial deviation between the enhancement coefficient and the second image, thereby reducing the inter-frame difference. Using the corrected enhancement coefficient to correct the contrast of the second image and replacing the second image with the second image with corrected contrast for display can improve the display effect.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image processing method and an electronic device. Background Art

[0002] The imaging quality of cameras provided on electronic devices, especially mobile phone cameras, is getting higher and higher. Since mobile phone cameras are convenient to use, more and more users pay more attention to the imaging quality of mobile phone cameras. For pictures or videos taken, users can adjust the imaging quality by adjusting the parameters of the pictures or videos, or use image editing software to process the pictures or videos to obtain a more satisfactory imaging quality. However, the imaging quality of the real-time preview image or the image directly presented to the user when shooting a video is also very important. Summary of the Invention

[0003] In view of this, an image processing method and an electronic device are provided. In image processing, the pixel offset of a subsequent frame image is predicted based on the current frame image, and the contrast of the image is corrected according to the pixel offset, which can reduce the inter-frame difference, solve the problem of video or image flicker caused by contrast enhancement in the related art, and improve the display effect.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method including: collecting a first image through a camera, where the first image is one of multiple frames of images continuously collected by the camera; predicting, based on the first image, the pixel offset of a second image after the first image in the multiple frames of images relative to the first image, where the pixel offset is the spatial position change of the pixels of the second image relative to the pixels of the first image; obtaining a first enhancement coefficient based on the first image and the pixel offset; collecting the second image through the camera; correcting the contrast of the second image according to the first enhancement coefficient; and displaying the corrected second image.

[0005] For the image processing method provided by the present application, during the process of continuously collecting images by the electronic device through the camera, the pixel offset of the subsequent second image relative to the already collected first image is predicted, the enhancement coefficient is corrected according to the pixel offset of the second image relative to the first image, and the contrast of the second image is corrected according to the corrected enhancement coefficient. By correcting the enhancement coefficient according to the spatial position change of the pixels, the spatial deviation between the enhancement coefficient and the image of the acting frame can be minimized, thereby reducing the inter-frame difference. Therefore, by correcting the contrast of the second image using the corrected enhancement coefficient and replacing the second image with the second image with corrected contrast for display, the display effect can be improved.

[0006] In the first possible implementation manner of the first aspect, predicting the pixel offset of a second image after the first image in the multi-frame images relative to the first image according to the first image includes: obtaining historical data of a motion sensor, and predicting the pixel offset of the second image relative to the first image according to the historical data of the motion sensor and the first image, where the historical data of the motion sensor is data before the second image is collected.

[0007] In the second possible implementation manner of the first aspect, predicting the pixel offset of a second image after the first image in the multi-frame images relative to the first image according to the first image includes: predicting the pixel offset of the second image relative to the first image according to the information of historical images before the second image and the first image.

[0008] In the third possible implementation manner of the first aspect, predicting the pixel offset of a second image after the first image in the multi-frame images relative to the first image according to the first image includes: predicting the pixel offset of the second image relative to the first image according to the historical data of the motion sensor, the information of historical images, and the first image before the moment when the second image is collected.

[0009] In the fourth possible implementation manner of the first possible implementation manner of the first aspect, obtaining historical data of a motion sensor and predicting the pixel offset of the second image relative to the first image according to the historical data of the motion sensor and the first image includes: obtaining historical data of a motion sensor, determining a motion scenario according to the historical data of the motion sensor; predicting the pixel offset of the second image relative to the first image according to the historical motion information corresponding to the motion scenario and the first image; where the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is the information of historical images.

[0010] By pre-judging the motion scenario and selecting appropriate historical motion information according to the motion scenario to predict the poses of future several frames of images, the prediction result can be made more accurate, so that the enhancement coefficient and the spatial deviation of the acting frames are minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0011] According to any one of the first aspect or the first to fourth possible implementation manners of the first aspect, in a fifth possible implementation manner, predicting a pixel offset of a second image after the first image in the multi-frame images relative to the first image according to the first image includes predicting a pose difference of the pose of the camera when acquiring the second image relative to the pose when acquiring the first image; and obtaining the pixel offset of the second image relative to the first image according to the pose difference.

[0012] According to the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner, the method further includes: if the historical pose prediction accuracy before the second image does not meet the accuracy condition, a process of adjusting the pose of a subsequent predicted image according to the pose difference, where the historical pose prediction accuracy is obtained according to the predicted pose difference and the corresponding actual pose difference before the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet. By adjusting the prediction process through the feedback of the prediction result, the prediction accuracy can be further improved, so that the enhancement coefficient and the spatial deviation of the action frame are minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0013] According to the seventh possible implementation manner of the first aspect, obtaining a first enhancement coefficient according to the first image and the pixel offset includes: calculating a second enhancement coefficient according to the first image; and correcting the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient. According to the above process, it can be seen that the image processing method provided in the embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by adjusting the enhancement coefficient according to the change in the spatial position of the pixels, thereby reducing the inter-frame difference and improving the enhancement effect.

[0014] According to the eighth possible implementation manner of the first aspect, obtaining a first enhancement coefficient according to the first image and the pixel offset includes: correcting the first image according to the pixel offset to obtain a third image; and calculating a first enhancement coefficient according to the third image. The image processing method provided in the embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by adjusting the first image according to the change in the spatial position of the pixels and calculating the enhancement coefficient according to the adjusted first image, thereby reducing the inter-frame difference and improving the enhancement effect.

[0015] Second aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: a camera, a processor, and a display. Among them, the camera is configured to collect a first image and send it to the processor; the first image is one of multiple frames of images continuously collected by the camera; the processor is configured to predict a pixel offset of a second image after the first image in the multiple frames of images relative to the first image according to the first image, where the pixel offset is a spatial position change of pixels of the second image relative to the first image; the processor is configured to obtain a first enhancement coefficient according to the first image and the pixel offset; the camera is configured to collect the second image and send it to the processor; the processor is configured to correct the contrast of the second image according to the first enhancement coefficient; the display is configured to display the corrected second image.

[0016] According to the electronic device provided by the embodiment of the present application, during the process of continuously collecting images by the camera, it predicts the pixel offset of the subsequent second image relative to the already collected first image, corrects the enhancement coefficient according to the pixel offset, and corrects the image contrast of the second image according to the corrected enhancement coefficient. The electronic device of the embodiment of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by correcting the enhancement coefficient according to the spatial position change of the pixels, thereby reducing the inter-frame difference. The contrast of the second image is corrected using the corrected enhancement coefficient, and the second image with the corrected contrast is used to replace the second image for display, which can improve the display effect.

[0017] According to the first possible implementation manner of the second aspect, the electronic device further includes a motion sensor, and the processor is configured to obtain historical data of the motion sensor and predict the pixel offset of the second image relative to the first image according to the historical data and the first image, where the historical data of the motion sensor is data before collecting the second image.

[0018] According to the second possible implementation manner of the second aspect, the processor is further configured to predict the pixel offset of the second image relative to the first image according to the information of the historical image before the second image and the first image.

[0019] According to the third possible implementation manner of the second aspect, the electronic device further includes a motion sensor, and the processor is further configured to predict the pixel offset of the second image relative to the first image according to the historical data of the motion sensor, the information of the historical image, and the first image before the moment of collecting the second image.

[0020] In the fourth possible implementation manner of the second aspect, the electronic device further includes a motion sensor, and the processor is further configured to obtain historical data of the motion sensor before collecting the second image, determine a motion scenario according to the historical data, and predict a pixel offset of the second image relative to the first image according to the historical motion information corresponding to the motion scenario and the first image; wherein, the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is the information of the historical image.

[0021] By predicting the poses of the next few frames of images according to the appropriate historical motion information selected according to the motion scenario, the prediction result can be made more accurate, so that the enhancement coefficient and the spatial deviation of the action frame are minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0022] According to any one of the second aspect or the first to fourth possible implementation manners of the second aspect, in the fifth possible implementation manner, predicting the pixel offset of the second image in the multiple frames of images relative to the first image according to the first image includes: predicting the pose difference of the pose of the camera when obtaining the second image relative to the pose when obtaining the first image; and obtaining the pixel offset of the second image relative to the first image according to the pose difference.

[0023] In the sixth possible implementation manner according to the fifth possible implementation manner of the second aspect, the processor is further configured to, when the historical pose prediction accuracy before the second image does not meet the accuracy condition, adjust the pose difference of the subsequent predicted image according to the pose difference, where the historical pose prediction accuracy is obtained according to the predicted pose difference and the corresponding actual pose difference before the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet. According to the above process, it can be known that the image processing device provided by the embodiments of the present application can further improve the prediction accuracy by adjusting the prediction process through the prediction result, so that the enhancement coefficient and the spatial deviation of the action frame are minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0024] In the seventh possible implementation manner of the second aspect, the processor is configured to calculate a second enhancement coefficient according to the first image, and correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient. According to the above process, it can be known that the image processing device provided by the embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by adjusting the enhancement coefficient according to the change in the spatial position of the pixel, thereby reducing the inter-frame difference and improving the enhancement effect.

[0025] In the eighth possible implementation manner of the second aspect, the processor is configured to correct the first image according to the pixel offset to obtain a third image, and calculate a first enhancement coefficient based on the third image. The image processing device provided in the embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by adjusting the first image according to the change in the spatial position of the pixels and calculating the enhancement coefficient based on the adjusted first image, thereby reducing the inter-frame difference and improving the enhancement effect.

[0026] In a third aspect, an embodiment of the present application provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs on an electronic device, the processor in the electronic device executes the image processing method according to one or several of the first aspect or the various possible implementation manners of the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides an image processing device, including: a processor; and a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions, it implements the image processing method according to one or several of the first aspect or the various possible implementation manners of the first aspect.

[0028] These and other aspects of the present application will become more apparent and understandable in the following description of the (multiple) embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings included in the specification and constituting a part of the specification, together with the specification, illustrate the exemplary embodiments, features, and aspects of the present application and are used to explain the principles of the present application.

[0030] Figure 1a A schematic diagram showing an application scenario of an image processing method according to an example of the present application.

[0031] Figure 1b An inter-frame timing diagram showing image enhancement processing according to an embodiment of the present application.

[0032] Figure 2a A schematic diagram showing the structure of an electronic device according to an embodiment of the present application.

[0033] Figure 2b It is a software structure block diagram of the electronic device in the embodiment of the present application.

[0034] Figure 3 An inter-frame timing diagram showing an image processing method according to an embodiment of the present application.

[0035] Figure 4 An example showing the calculation of an enhancement coefficient according to an embodiment of the present application.

[0036] Figure 5a It is a schematic diagram for visually representing the relationship between the enhancement coefficient and the position of pixels in an image.

[0037] Figure 5b It shows a schematic diagram of trilinear interpolation.

[0038] Figure 6a It shows a schematic diagram of the pose and pose difference of a camera according to an embodiment of the present application.

[0039] Figure 6b It shows a schematic diagram of the camera motion and pose prediction according to an embodiment of the present application.

[0040] Figure 7 It shows a schematic diagram of the pixel offset of a second image relative to a first image according to an embodiment of the present application.

[0041] Figure 8 It shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application.

[0042] Figure 9a It shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application.

[0043] Figure 9b It shows a schematic diagram of target motion trajectory prediction according to an embodiment of the present application.

[0044] Figure 9c It shows a schematic diagram of the pixel offset of a second image relative to a first image according to an embodiment of the present application.

[0045] Figure 10 It shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application.

[0046] Figure 11a It shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application.

[0047] Figure 11b It shows a schematic diagram of determining the historical pose prediction accuracy according to an embodiment of the present application.

[0048] Figure 12 It shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application.

[0049] Figure 13 It shows a flowchart of an image processing method according to an embodiment of the present application.

[0050] Figure 14 It shows a block diagram of an image processing apparatus according to an embodiment of the present application.

[0051] Figure 15A schematic diagram showing an application scenario according to an embodiment of the present application.

[0052] Figure 16 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0053] Figure 17 A schematic diagram showing an application scenario according to an embodiment of the present application. Detailed implementation manners

[0054] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0055] The special term "exemplary" here means "serving as an example, embodiment, or illustrative". Any embodiment described as "exemplary" here is not necessarily to be construed as superior to or better than other embodiments.

[0056] In addition, for a better description of the present application, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present application can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail so as to highlight the gist of the present application.

[0057] Figure 1a A schematic diagram showing an application scenario of an image processing method according to an embodiment of the present application. Figure 1a What is shown may be a lens, an image sensor, and an image processor of a camera in an electronic device. The electronic device according to the embodiment of the present application may be a camera, a smart phone, a netbook, a tablet computer, a notebook computer, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a TV, a virtual reality device, a speaker, an electronic ink, and so on. The electronic device according to the embodiment of the present application is integrated with a lens, an image sensor, an image processor, a display, etc., for collecting an image, processing the image, and displaying the processed image on the display.

[0058] Such as Figure 1aAs shown in the figure, after the lens projects the optical signal onto the image sensor, the image sensor performs photoelectric conversion on the optical signal to obtain an electrical signal (image signal), and sends the electrical signal to the image processor for processing. Among them, the image sensor can be a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge-coupled Device (CCD) sensor. The present application does not limit the specific type of the image sensor. The image processor can be a dedicated image processing engine that is pipelined: an Image Signal Processor (ISP), which can process image signals at high speed.

[0059] In order to ensure the contrast of the image, the image signal processor can use a contrast enhancement algorithm to process the received image signal. Image contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in the bright and dark regions of an image, that is, the size of the gray-scale contrast of an image. The larger the difference range, the greater the contrast; the smaller the difference range, the smaller the contrast. The contrast of an image can be understood as the degree to which details in the image can be distinguished by the naked eye. The contrast can be reflected by the histogram of the image. The color histogram distribution of an image with small contrast often distributes within a relatively small pixel value range, while the color histogram of a clearer (higher-contrast) image is more evenly distributed within the entire pixel gray value range. Therefore, the higher the image contrast, the greater the horizontal span and the more consistent the vertical length in the rectangular coordinate system composed of gray scale / pixel count. The longer the span on the abscissa indicates that more gray values are used in the image, and thus it can more clearly reflect the difference between each detail in the image and the details around it. The global contrast of the image is the above-mentioned image contrast, that is, the size of the gray-scale contrast of the image. The local contrast of the image refers to the contrast of the image in a partial area of the image.

[0060] The contrast enhancement algorithm can enhance the image specifically according to statistical information such as the brightness and color of the image, but the statistical information generally lags behind the image being processed, especially in scenarios where the camera is previewing or shooting a video.

[0061] In a possible implementation manner, enhancing the contrast of the image according to the statistical information of the image may include: calculating an enhancement coefficient for correcting the contrast of the image according to the statistical information of the image, and correcting the contrast of the image according to the enhancement coefficient. In the embodiments of the present application, the enhancement coefficient represents the adjustment coefficient of the pixel value of the pixels in the image. Therefore, the enhancement coefficient obtained according to the statistical information also lags behind the image being processed.

[0062] For example, during video shooting or in previewed images, there is a pixel offset between different frames, and the pixel offset indicates a change in the spatial position of some or all pixels in different images. For example, in one scenario, during image preview or video shooting, if the camera remains stationary, but there are moving targets (such as moving vehicles, pedestrians, etc.) within the framing range, the local areas between images with adjacent or similar acquisition times are different. This difference may be manifested as the spatial position of some pixels in the entire image moving, while other information (such as color, brightness, etc.) has not changed, and there is a pixel offset between adjacent images. In another scenario, if the target within the framing range is stationary, but the camera is continuously moving, the position of the pixels between images with adjacent or similar acquisition times relative to the boundary of the framing range changes, that is, the spatial position of most pixels in the image has moved within the entire image, and there is a pixel offset.

[0063] Since the enhancement coefficients correspond to pixels at different positions in the image, that is, the enhancement coefficients calculated based on the image correspond to the pixels in the image. If the position of the pixels is offset, there will be a spatial deviation between the enhancement coefficients and the processed image. Using the enhancement coefficients to process the image will introduce instability between frames, causing flickering in the video or preview image. The flickering is mainly manifested as changes in brightness and contrast in some areas of the image.

[0064] Figure 1b FIG. 2 shows a timing diagram of an image enhancement process performed according to an embodiment of the present application. Figure 1b As shown, the enhancement coefficient corresponding to the pixel in the image is calculated based on the in-th frame image. When the processor processes the image based on the calculated enhancement coefficient, the image processed is the ith frame image. The ith frame image may have a pixel offset relative to the in-th frame image. Using the enhancement coefficient to process the image will introduce instability between frames, causing the video or preview image to flicker.

[0065] In addition, the processor of the electronic device uses a multi-task parallel processing mode, and each sub-thread occupies the resources of the central processing unit (CPU), so the performance of the CPU will fluctuate. The software scheduling process of the ISP also has performance fluctuations, which will lead to fluctuations in the frame delay of the enhancement coefficient. The frame delay of the enhancement coefficient can refer to the time difference between the frame corresponding to the statistical information of the enhancement coefficient calculated and the frame using the enhancement coefficient for contrast enhancement. The fluctuation of the frame delay of the enhancement coefficient will cause unpredictable flickering of the video or preview image.

[0066] To solve the above technical problems, the present application provides an image processing method, which can be applied to an electronic device. Taking the electronic device as a mobile phone as an example, Figure 2a Shows a schematic structural diagram of an electronic device according to an embodiment of the present application. Figure 2a Shows a schematic structural diagram of the mobile phone 200. The electronic device of the present application is not limited to a mobile phone. As described above, it can also be other types of electronic devices.

[0067] Such as Figure 2a As shown, the mobile phone 200 may include a processor 210, an external memory interface 220, an internal memory 221, a USB interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 251, a wireless communication module 252, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc. Among them, the sensor module 280 may include a gyroscope sensor 280A, an acceleration sensor 280B, a proximity light sensor 280G, a fingerprint sensor 280H, a touch sensor 280K (of course, the mobile phone 200 may also include other sensors, such as a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, a barometric pressure sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0068] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the mobile phone 200. In other embodiments of the present application, the mobile phone 200 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. Among them, the controller may be the nerve center and command center of the mobile phone 200. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0070] A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory may save the instructions or data that the processor 210 has just used or recycled. If the processor 210 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0071] The processor 210 may run the image processing method provided in the embodiments of the present application to minimize the enhancement coefficient and the spatial deviation of the action frame, thereby reducing the inter-frame difference and improving the enhancement effect. The processor 210 may include different devices. For example, when the CPU and GPU are integrated, the CPU and GPU may cooperate to execute the image processing method provided in the embodiments of the present application. For example, some algorithms in the image processing method are executed by the CPU, and another part of the algorithms are executed by the GPU to obtain a faster processing efficiency.

[0072] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0073] In some embodiments, the mobile phone 200 may include one or N display screens 294, where N is a positive integer greater than 1. The display screen 294 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs). For example, the display 294 can display photos, videos, web pages, or files, etc. For another example, the display 294 can display a graphical user interface. Among them, the graphical user interface includes a status bar, a hidden navigation bar, a time and weather widget, and application icons, such as a browser icon, etc. The status bar includes the operator name (such as China Mobile), the mobile network (such as 4G), the time, and the remaining battery power. The navigation bar includes a back key icon, a home key icon, and a forward key icon. In addition, it can be understood that in some embodiments, the status bar may also include a Bluetooth icon, a Wi-Fi icon, an external device icon, etc. It can also be understood that in some other embodiments, the graphical user interface may further include a Dock bar, and the Dock bar may include common application icons, etc. When the processor 210 detects a touch event of the user's finger (or a stylus, etc.) on a certain application icon, in response to the touch event, it opens the user interface of the application corresponding to the application icon and displays the user interface of the application on the display 294.

[0074] In the embodiments of the present application, the display screen 294 can be an integrated flexible display screen, or a spliced display screen composed of two rigid screens and a flexible screen located between the two rigid screens. In the embodiments of the present application, the display screen 294 can display the image after correcting the contrast.

[0075] The camera 293 (front camera or rear camera, or a single camera that can function as both a front camera and a rear camera, and both the front camera and the rear camera can include multiple cameras) is used to capture static images or videos. Generally, the camera 293 may include photosensitive elements such as a lens group and an image sensor. Among them, the lens group may include the lenses in the scene shown in this application Figure 1a The lenses in the scene shown in this application. The lens group includes multiple lenses (convex lenses or concave lenses) for collecting the optical signals reflected by the object to be photographed and transmitting the collected optical signals to the image sensor. The image sensor generates the original image (electrical signal) of the object to be photographed according to the optical signal. The processed image can be displayed on the display screen 294. When the processor 210 runs the image processing method provided in the embodiments of this application, the electronic device can display the processed image on the display screen 294.

[0076] The internal memory 221 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 210 executes various functional applications and data processing of the mobile phone 200 by running the instructions stored in the internal memory 221. The internal memory 221 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, the codes of application programs (such as camera applications, WeChat applications, etc.). The data storage area can store the data created during the use of the mobile phone 200 (such as images, videos, etc. collected by the camera application).

[0077] The internal memory 221 can also store one or more computer programs 1310 corresponding to the image processing method provided in the embodiments of this application. The one or more computer programs 1304 are stored in the above-mentioned memory 221 and are configured to be executed by the one or more processors 210. The one or more computer programs 1310 include instructions, and the above instructions can be used to execute the image processing method provided in the embodiments of this application.

[0078] In addition, the internal memory 221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0079] Of course, the code of the image processing method provided in the embodiments of this application can also be stored in an external memory. In this case, the processor 210 can run the code of the image processing method stored in the external memory through the external memory interface 220.

[0080] Next, the functions of the sensor module 280 will be introduced.

[0081] The gyroscope sensor 280A can be used to determine the motion posture of the mobile phone 200. In some embodiments, the angular velocity of the mobile phone 200 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 280A. That is, the gyroscope sensor 280A can be used to detect the current motion state of the mobile phone 200, such as whether it is shaking or stationary.

[0082] When the display screen in the embodiments of the present application is a foldable screen, the gyroscope sensor 280A can be used to detect the folding or unfolding operation on the display screen 294. The gyroscope sensor 280A can report the detected folding operation or unfolding operation as an event to the processor 210 to determine the folding state or unfolding state of the display screen 294.

[0083] The acceleration sensor 280B can detect the magnitude of the acceleration of the mobile phone 200 in various directions (generally three axes). When the display screen in the embodiments of the present application is a foldable screen, the acceleration sensor 280B can be used to detect the folding or unfolding operation on the display screen 294. The acceleration sensor 280B can report the detected folding operation or unfolding operation as an event to the processor 210 to determine the folding state or unfolding state of the display screen 294.

[0084] In some embodiments, in the image processing method of the present application, the motion path of the camera can be estimated by motion sensors such as the gyroscope sensor and the acceleration sensor, so as to predict the pose of the camera.

[0085] The gyroscope sensor 280A (or the acceleration sensor 280B) can send the detected motion state information (such as angular velocity) to the processor 210. The processor 210 determines whether it is a handheld state or a tripod state based on the motion state information (for example, when the angular velocity is not 0, it means that the mobile phone 200 is in the handheld state). The processor 210 can also judge the motion scene according to the motion state information, predict the future pose of the camera, and so on. The proximity light sensor 280G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The mobile phone emits infrared light outward through the light-emitting diode. The mobile phone uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone. When insufficient reflected light is detected, the mobile phone can determine that there is no object near the mobile phone. When the display screen in the embodiments of the present application is a foldable screen, the proximity light sensor 280G can be disposed on the first screen of the foldable display screen 294, and the proximity light sensor 280G can detect the folding angle or unfolding angle between the first screen and the second screen according to the optical path difference of the infrared signal.

[0086] The fingerprint sensor 280H is used to collect fingerprints. The mobile phone 200 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to the application lock, fingerprint photography, fingerprint answering of incoming calls, etc.

[0087] The touch sensor 280K, also known as the "touch panel". The touch sensor 280K can be disposed on the display screen 294. The touch sensor 280K and the display screen 294 form a touch screen, also known as a "touch control screen". The touch sensor 280K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In some other embodiments, the touch sensor 280K can also be disposed on the surface of the mobile phone 200, at a different position from that of the display screen 294.

[0088] Exemplarily, the display screen 294 of the mobile phone 200 displays the main interface, and the main interface includes icons of multiple applications (such as a camera application, a WeChat application, etc.). The user clicks the icon of the camera application in the main interface through the touch sensor 280K, triggering the processor 210 to start the camera application and turn on the camera 293. The display screen 294 displays the interface of the camera application, such as a viewfinder interface.

[0089] The wireless communication function of the mobile phone 200 can be implemented by the antenna 1, the antenna 2, the mobile communication module 251, the wireless communication module 252, the modulation and demodulation processor, and the baseband processor, etc.

[0090] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0091] The mobile communication module 251 can provide wireless communication solutions including 2G / 3G / 4G / 5G, etc. applied to the mobile phone 200. The mobile communication module 251 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 251 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 251 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 251 can be disposed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 251 and at least some modules of the processor 210 can be disposed in the same device.

[0092] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 270A, the receiver 270B, etc.), or displays an image or video through the display screen 294. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 210 and be provided in the same device as the mobile communication module 251 or other functional modules.

[0093] The wireless communication module 252 may provide solutions for wireless communications applied to the mobile phone 200, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 252 may be one or more devices integrating at least one communication processing module. The wireless communication module 252 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 210. The wireless communication module 252 may also receive the signal to be transmitted from the processor 210, perform frequency modulation and amplification on it, and convert it into electromagnetic waves through the antenna 2 and radiate it out. In the embodiments of the present application, the wireless communication module 252 is used to transmit data to and from other electronic devices under the control of the processor 210.

[0094] In addition, the mobile phone 200 can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor, etc. For example, music playback, recording, etc. The mobile phone 200 can receive inputs from the keys 290 and generate key signal inputs related to the user settings and function controls of the mobile phone 200. The mobile phone 200 can use the motor 291 to generate vibration prompts (such as incoming call vibration prompts). The indicator 292 in the mobile phone 200 can be an indicator light, which can be used to indicate the charging state, the change in battery power, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 295 in the mobile phone 200 is used to connect the SIM card. The SIM card can be in contact with and separated from the mobile phone 200 by being inserted into or removed from the SIM card interface 295.

[0095] It should be understood that in actual applications, the mobile phone 200 may include more or fewer components than Figure 2a those shown. The embodiments of the present application do not make any limitations. The illustrated mobile phone 200 is only an example, and the mobile phone 200 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0096] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. Taking the Android system with a layered architecture as an example in the embodiments of the present application, the software structure of the electronic device is illustratively described.

[0097] Figure 2b is the software structure block diagram of the electronic device in the embodiments of the present application.

[0098] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the Android runtime and the system libraries, and the kernel layer.

[0099] The application layer may include a series of application packages.

[0100] As Figure 2b shown, the application packages may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0101] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0102] As Figure 2b shown, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0103] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0104] The content provider is used to store and obtain data, and enable these data to be accessed by applications. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0105] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0106] The telephone manager is used to provide the communication function of the electronic device. For example, the management of call status (including connection, hanging up, etc.).

[0107] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0108] The notification manager enables applications to display notification information in the status bar, can be used to convey notification-type messages, can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scroll bar text, such as a notification of a background running application, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompting text information in the status bar, emitting a prompt sound, vibrating the electronic device, flashing the indicator light, etc.

[0109] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0110] The core library contains two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android.

[0111] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and the application framework layer as binary files. The virtual machine is used to manage the object life cycle, stack management, thread management, security and exception management, and garbage collection and other functions.

[0112] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0113] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0114] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0115] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0116] The 2D graphics engine is a drawing engine for 2D drawing.

[0117] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0118] The above is an introduction to the electronic device in the application scenario of the image processing method of the present application. Next, in combination with Figure 2a and Figure 2b The application scenario shown is used to illustrate the image processing method of the present application.

[0119] In the image processing method of the embodiment of the present application, by predicting the pixel offset of the future frame of the image relative to the current frame and correcting the enhancement coefficient according to the predicted pixel offset, the spatial deviation between the enhancement coefficient and the corresponding pixel points in the future frame is minimized, thereby reducing the inter-frame difference and improving the enhancement effect.

[0120] In an embodiment of the present application, the electronic device may execute the image processing method provided by the present application during image preview or video shooting. For example, taking a mobile phone as an example, when a user holds the mobile phone to take a picture, the user previews the image collected by the camera on the display screen before taking the picture, or the user holds the mobile phone to shoot a video. In other examples, the camera may also be a camera installed on a drone, and the drone shoots while flying; or, the camera may also be a vehicle-mounted camera (dash cam), and when the vehicle is moving, the camera shoots the scenes around the vehicle, and so on. The present application does not limit the specific application scenarios, and the camera can be during continuous viewfinder or shooting.

[0121] During image preview or video shooting, if the camera remains stationary, but there are moving targets (such as moving vehicles, pedestrians, etc.) within the viewfinder, the local areas between images with adjacent or close acquisition times are different, that is, there is local motion. The later acquired image may have a spatial position change of some pixels relative to the earlier acquired image, that is, there is pixel offset. If the target within the viewfinder is stationary, but the camera is continuously moving, the position of the pixels relative to the boundary of the viewfinder range changes between images with adjacent or close acquisition times, that is, there is global motion. The later acquired image may have a spatial position change of pixels relative to the earlier acquired image, that is, there is pixel offset. If the camera is continuously moving and there are also moving targets within the viewfinder, then there is both local motion and global motion between images with adjacent or close acquisition times. The later acquired image may have a spatial position change of pixels relative to the earlier acquired image, that is, there is pixel offset.

[0122] Specifically, the electronic device may collect a first image through the camera, where the first image is one of multiple frames of images continuously collected by the camera; based on the first image, the electronic device may predict the pixel offset of a second image after the first image among the multiple frames of images relative to the first image, where the pixel offset is the spatial position change of the pixels of the second image relative to the first image; the electronic device may obtain a first enhancement coefficient based on the first image and the pixel offset; the electronic device may collect the second image through the camera and correct the contrast of the second image according to the first enhancement coefficient; the electronic device may display the corrected second image, that is, display the corrected second image instead of the second image.

[0123] In an embodiment of the present application, the camera may be a device including a lens group and an image sensor, such as Figure 2aFor the camera 293 shown, the lens group can collect the optical signals reflected by the object to be photographed and transmit the collected optical signals to the image sensor, and the image sensor generates an image of the object to be photographed according to the optical signals. This image can be the above-mentioned first image or second image.

[0124] In the embodiments of the present application, the electronic device can predict the pixel offset of the future frame relative to the previous frame by means of moving object detection, motion estimation, pose monitoring of the camera, pose estimation, etc. Specifically, the electronic device can predict the pose difference of the camera at the moment of obtaining the second image relative to the pose at the moment of obtaining the first image, and obtain the pixel offset of the second image relative to the first image according to the pose difference. In a possible implementation manner, the electronic device can predict the poses of the camera when obtaining multiple frames of images after obtaining the first image, and predict the number of delayed frames of the second image relative to the first image; the electronic device can determine the pose difference of the camera at the moment of obtaining the second image relative to the pose at the moment of obtaining the first image according to the predicted poses of the camera when obtaining multiple frames of images after obtaining the first image and the number of delayed frames; the electronic device can obtain the pixel offset of the second image relative to the first image according to the pose difference. For example, the pose difference can be converted from the world coordinate system to the image plane coordinate system to obtain the pixel offset.

[0125] In the embodiments of the present application, the pose or pose difference of the camera predicted by the electronic device is the pose or pose difference of the above-mentioned camera when collecting images.

[0126] In an example, the electronic device can calculate a second enhancement coefficient according to the first image, correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient. In another example, the electronic device can correct the first image according to the pixel offset to obtain a third image, and calculate the first enhancement coefficient according to the third image. That is to say, the processor can first calculate the enhancement coefficient and correct the enhancement coefficient according to the pixel offset, or can first correct the image according to the pixel offset and then calculate the enhancement coefficient according to the corrected image. The present application does not limit this.

[0127] In a possible implementation manner, the electronic device corrects the second enhancement coefficient according to the pixel offset, which can be to correct the relationship between the second enhancement coefficient and the corresponding pixel. Correcting the image information of the first image according to the pixel offset can be to correct the position coordinates of the pixels in the first image so that the adjusted position coordinates are the same as the predicted ones, so as to minimize the inter-frame difference and improve the enhancement effect.

[0128] The process of calculating the enhancement coefficient of an electronic device can adopt the methods in related technologies. For example, statistical information can be obtained based on the image information of the first image. The image information can be information such as the brightness, color, and grayscale of the image. For example, the grayscale distribution of the first image can be statistically analyzed, and the enhancement coefficient corresponding to the first image can be calculated based on the statistical information. Specifically, the electronic device can calculate the enhancement coefficient corresponding to a pixel based on the statistical information and the image information of the pixel in the first image. The electronic device can calculate the enhancement coefficient corresponding to each pixel. Or, in order to improve the processing speed, the first image can also be partitioned, the pixels in each region are statistically analyzed to obtain the statistical information of each region, and the enhancement coefficient corresponding to the pixel at the intersection point (vertex) of the region can be calculated based on the statistical information of each region. This application does not limit the specific process of calculating the enhancement coefficient.

[0129] In the image processing method provided by this application, during the process of an electronic device continuously acquiring images through a camera, it predicts the pixel offset of the subsequent second image relative to the already acquired first image, corrects the enhancement coefficient according to the pixel offset of the second image relative to the first image, and corrects the image contrast of the second image according to the corrected enhancement coefficient. By correcting the enhancement coefficient according to the change in the spatial position of the pixels, the spatial deviation between the enhancement coefficient and the image of the frame being processed can be minimized, thereby reducing the inter-frame difference. Therefore, using the corrected enhancement coefficient to correct the contrast of the second image and replacing the second image with the second image with the corrected contrast for display can improve the display effect.

[0130] Figure 3 The inter-frame timing diagram showing the image processing method according to an embodiment of the present application is as follows Figure 3 As shown, it is assumed that the (i - n)-th frame image represents the above-mentioned first image, and the i-th frame image represents the second image.

[0131] In Figure 3 In the shown example, the electronic device can calculate the second enhancement coefficient through the camera acquiring the first image. The electronic device can calculate the second enhancement coefficient based on the image information of the first image (the (i - n)-th frame image).

[0132] Figure 4 The example showing the calculation of the enhancement coefficient according to an embodiment of the present application is as follows Figure 4 As shown, it can be an image acquired by the electronic device through the camera. The electronic device can divide the pixels on the image into multiple different regions in the width (W, Width) and height (H, High) directions. Figure 4A three-dimensional coordinate system (W, H, L) is established with the lower left corner of the middle image as the coordinate origin O. Among them, W represents the coordinate axis in the width direction, H represents the coordinate axis in the height direction, and L represents the coordinate axis in the brightness direction. In the embodiments of the present application, the coordinates in the W and H directions can be called position coordinates, and the coordinates in the L direction can be called brightness coordinates. Figure 4 a, b, c, d, and e in it respectively represent pixel points in the image. Among them, the coordinates of pixel a are (wa, ha, la), where wa represents the position of pixel a in the W direction, ha represents the position of pixel a in the H direction, and la represents the brightness of pixel a.

[0133] In Figure 4 In the shown image, the pixels at the vertices of each region can correspond to one or more enhancement coefficients. In the embodiments of the present application, the pixels in a frame of image can be divided into two categories. One category of pixels is located at the intersection points (vertices of the regions) of the horizontal dividing line and the vertical dividing line. In the embodiments of the present application, this category of pixels is called the first category of pixels, and the pixels other than the first category of pixels on the image can be called the second category of pixels. When calculating the enhancement coefficient, the electronic device can obtain the statistical information of the pixels in the region according to the image information of the pixels in each region, calculate the enhancement coefficient of the first category of pixels corresponding to the region according to the statistical information of the pixels in the region, and the enhancement coefficient corresponding to the second category of pixels can be obtained according to the enhancement coefficient corresponding to the first category of pixels adjacent to the second category of pixels. Among them, the first category of pixels adjacent to the second category of pixels can refer to the first category of pixels corresponding to the vertices of the region where the second category of pixels is located.

[0134] For example, the display resolution of the display screen of the electronic device is 1280×720. According to Figure 4 In the shown example, it is divided into four regions in both the W and H directions. In the left-to-right direction, the W1 region includes the 1st - 320th column of pixels, the W2 region includes the 321st - 640th column of pixels, the W3 region includes the 641st - 960th column of pixels, and the W4 region includes the 961st - 1280th column of pixels. In the bottom-to-top direction, the H1 region includes the 1st - 180th row of pixels, the H2 region includes the 181st - 360th row of pixels, the H3 region includes the 361st - 480th row of pixels, and the H4 region includes the 481st - 720th row of pixels. The pixel at the 1st row and 1st column is the first category of pixels, the pixel at the 1st row and 321st column is the first category of pixels, the pixel at the 1st row and 641st column is the first category of pixels..., the pixel at the 181st row and 1st column is the first category of pixels, the pixel at the 361st row and 1st column is the first category of pixels..., and other pixels are the second category of pixels.

[0135] In a possible implementation, the pixel value of a pixel can be represented by (r, g, b, l), where r (red), g (green), b (blue), and l (light) can respectively represent the red, green, blue, and brightness values of the pixel, and the enhancement coefficient can be a 2D matrix of 3×4. It should be noted that the above is only one representation of the enhancement coefficient, and the enhancement coefficient can also be data represented in other forms, which is not limited in this application.

[0136] Figure 5a is a schematic diagram for intuitively representing the relationship between the enhancement coefficient and the position of pixels in the image. Figure 5a In the three-dimensional model shown, the same Figure 4 coordinate system is adopted, and in the width (W, Width) direction and height (H, High) direction, respectively, Figure 4 the same method of dividing regions is adopted, that is Figure 4 and Figure 5a the method of dividing regions in the W direction is the same, the method of dividing regions in the H direction is the same, and the method of dividing regions in the L direction is also the same. Figure 5a In [the figure], a, b, c, and d respectively correspond to Figure 4 the pixels a, b, c, and d in [the figure]. Along the L direction, they respectively represent the enhancement coefficients corresponding to different brightnesses. At the intersection points of each region in Figure 5a corresponds an enhancement coefficient (matrix).

[0137] Figure 5a The L (brightness, Light) direction is also divided into multiple regions, that is, for different brightnesses corresponding to a position in the image, multiple enhancement coefficients corresponding to different brightnesses are set. In the embodiments of this application, multiple enhancement coefficients corresponding to different brightnesses can be set for Figure 4 the pixel points (the first type of pixel points) located at the intersection points of the regions. Taking the above example as an example, the enhancement coefficient is determined for the pixel according to the statistical information of the pixels in the region where the pixel is located. According to the statistical information of the pixels in the W1H1 region, multiple different enhancement coefficients are determined for the pixel located in the first row and the first column. According to the statistical information of the pixels in the W2H1 region, multiple different enhancement coefficients are determined for the pixel located in the first row and the 321st column. According to the statistical information of the pixels in the W3H1 region, multiple different enhancement coefficients are determined for the pixel located in the first row and the 641st column... According to the statistical information of the pixels in the W1H2 region, multiple different enhancement coefficients are determined for the pixel located in the 181st row and the first column..., that is Figure 4 in the attached drawings shown, for each pixel point located at the intersection point, there are corresponding multiple different enhancement coefficients, and the multiple different enhancement coefficients respectively correspond to different brightnesses. As Figure 4 and Figure 5a shown, for example, for Figure 4Pixel point a in Figure 5a The point a in corresponds to 5 different enhancement coefficients (matrices). Figure 5a The position of each intersection point shown corresponds to an enhancement coefficient (matrix), that is Figure 5a Each element in the three-dimensional model shown is an enhancement coefficient (matrix).

[0138] For Figure 4 Pixel e in, the intersection points adjacent to pixel e are pixel a, pixel b, pixel c, and pixel d. The electronic device can determine the position of pixel e in Figure 5a according to the position and brightness of pixel e. Assume that along the L direction from bottom to top, the enhancement coefficients corresponding to pixel a are a1, a2, a3, a4, a5 respectively, the enhancement coefficients corresponding to pixel b are b1, b2, b3, b4, b5 respectively, the enhancement coefficients corresponding to pixel c are c1, c2, c3, c4, c5 respectively, and the enhancement coefficients corresponding to pixel d are d1, d2, d3, d4, d5 respectively. The electronic device can determine which matrix body pixel e is located in Figure 5a according to the brightness and position of pixel e. The electronic device can determine the enhancement coefficient corresponding to pixel e according to the eight vertices of the matrix body where the position of pixel e in Figure 5a is located.

[0139] In a possible implementation, the electronic device can establish a coordinate system (such as Figure 4 and Figure 5a shown in the example, the image plane coordinate system can be adopted and the brightness coordinate can be added) and a look-up table. Through the coordinate system and the look-up table, the mapping relationship between the pixels in the figure and the enhancement coefficients can be established. Assume that Figure 4 the lower left corner of the image in is used as the coordinate origin O to establish a three-dimensional coordinate system (W, H, L). The coordinate of pixel a is (wa, ha, la), where wa represents the position of pixel a in the W direction, ha represents the position of pixel a in the H direction, and la represents the brightness of pixel a. In the embodiments of the present application, the electronic device calculates the second enhancement coefficient according to the first image, or calculates the first enhancement coefficient according to the third image. Both the first enhancement coefficient and the second enhancement coefficient can include the corresponding relationship between the coordinates of the pixels and the enhancement coefficients corresponding to the pixels. For example, the electronic device may store a first look-up table and / or a second look-up table. The first look-up table may record the corresponding relationship between the position coordinates of the pixels and the second look-up table, as shown in Table 1. The second look-up table may record the corresponding relationship between the brightness of the pixels and the enhancement coefficients of the pixels, as shown in Table 2.

[0140] In an embodiment of the present application, the electronic device may store the second enhancement coefficient in the memory. When correcting the second enhancement coefficient according to the pixel offset, the electronic device may read the second enhancement coefficient from the memory for correction to obtain the first enhancement coefficient, and store the first enhancement coefficient in the memory. Alternatively, the electronic device may store the first enhancement coefficient calculated based on the third image in the memory.

[0141] In an embodiment of the present application, when the electronic device corrects the contrast of the second image according to the first enhancement coefficient, the electronic device may read the first enhancement coefficient from the memory and correct the contrast of the second image according to the first enhancement coefficient.

[0142] It should be noted that storing the first lookup table and the second lookup table is only an example for the present application to record the pixels and the corresponding enhancement coefficients. The present application is not limited thereto. For example, it may also be recorded through a single table or by other means.

[0143] Table 1

[0144] w1 w2 …… wn h1 Table 11 Table 12 Table 1n h2 Table 21 Table 22 Table 2n …… hm Table m1 Table m2 …… Table mn

[0145] Table 2

[0146]

[0147] As shown in Table 1 and Table 2, when correcting the contrast of an image, for the first type of pixels, the second lookup table corresponding to the pixels may be determined according to the position coordinates w and h of the first type of pixels in the image and the first lookup table, and the enhancement coefficient corresponding to the pixels may be determined according to the second lookup table and the luminance coordinates of the first type of pixels.

[0148] After determining the enhancement coefficient corresponding to the pixels, the contrast may be corrected according to the pixel value and the enhancement coefficient of the pixels. In one possible implementation, the corrected pixel value may be calculated according to the following formula (1):

[0149]

[0150] Among them, the matrix may represent the enhancement coefficient, r, g, b, and l may respectively represent the pixel values before contrast correction of the pixels, and R, G, and B may respectively represent the pixel values after contrast correction of the pixels.

[0151] For example, for Figure 4The pixel a shown has a position coordinate of (w2, h1) in the i-nth frame image. After calculating the enhancement coefficients based on the i-nth frame image to obtain the above-mentioned first query table and second query table, if the image contrast is to be corrected, theoretically, in the i-nth frame image, pixel a is a first-type pixel. If the second query table corresponding to pixel a is to be obtained by querying the first query table according to the position coordinate in the i-nth frame image, it should be Table 12. According to Table 12 and the brightness of pixel a, the enhancement coefficient corresponding to pixel a can be determined. According to the pixel value and the corresponding enhancement coefficient, the enhanced pixel value can be calculated according to formula (1).

[0152] In the embodiments of the present application, the enhancement coefficient corresponding to the second-type pixel can be calculated based on the brightness of the second-type pixel and the enhancement coefficient corresponding to the first-type pixel adjacent to the second-type pixel. For example, Figure 4 When the pixel falls at a position outside the intersection point, the position of the pixel in the Figure 5a shown three-dimensional model can be determined according to the brightness and position of the pixel, and the enhancement coefficient corresponding to the pixel can be calculated according to the position of the pixel in the three-dimensional model and the enhancement coefficients corresponding to the 8 vertices of the region at that position.

[0153] For example, as Figure 4 the pixel e in, the intersection points adjacent to pixel e are pixel a, pixel b, pixel c, and pixel d. The electronic device can determine the position of pixel e in the Figure 5a according to the position and brightness of pixel e.

[0154] The brightness of pixel e can be calculated according to the pixel value (r, g, b): el (brightness of pixel e) = (0.299 * r) + (0.587 * g) + (0.114 * b). el can also be obtained by e passing through a function or filtering or network learning, and the general expression is el = f(……ri, gi, bi,……), where ri, gi, bi are the pixel values within the neighborhood of e. The pixels within the neighborhood of pixel e can refer to the pixels within a certain range around pixel e, and the size of the neighborhood can be determined according to actual needs, such as the adjustment accuracy, calculation speed, etc. The present application does not limit this.

[0155] As described above, along the L direction from bottom to top, the enhancement coefficients corresponding to pixel a are a1, a2, a3, a4, a5 respectively, the enhancement coefficients corresponding to pixel b are b1, b2, b3, b4, b5 respectively, the enhancement coefficients corresponding to pixel c are c1, c2, c3, c4, c5 respectively, and the enhancement coefficients corresponding to pixel d are d1, d2, d3, d4, d5 respectively. Assume that the electronic device determines the position of pixel e in the Figure 5aThe position in is within a rectangle formed by eight points a1, a2, b1, b2, c1, c2, d1, and d2. The electronic device can perform trilinear interpolation based on a1, a2, b1, b2, c1, c2, d1, and d2 to obtain the enhancement coefficient corresponding to pixel e.

[0156] Figure 5b shows a schematic diagram of trilinear interpolation. As Figure 5b shown, pixel e falls within the rectangle formed by eight points a1, a2, b1, b2, c1, c2, d1, and d2. Assume that the distance between pixel e and the plane a1a2c1c2 is tw, the distance between pixel e and the plane a2b2d2c2 is tl, and the distance between pixel e and the plane c1c2d1d2 is th.

[0157] Figure 5a The luminance value corresponding to the boundary (interface) of each region along the L direction in is fixed. For example, the range of the luminance value is 0 to 255. The luminance value corresponding to the plane a1b1d1c1 is 0, the luminance value corresponding to the plane a2b2d2c2 is 63.75, the luminance value corresponding to the plane a3b3d3c3 is 127.5, the luminance value corresponding to the plane a4b4d4c4 is 191.25, and the luminance value corresponding to the plane a5b5d5c5 is 255.

[0158] The luminance of pixel e can be calculated based on the pixel values (r, g, b). Based on the luminance and position of pixel e, the corresponding position of pixel e in Figure 5a can be determined. As Figure 4 and Figure 5b shown, the distance tw between pixel e and the plane a1a2c1c2 can be determined by the distance between pixel e and the side ac in Figure 4 . tw is equal to the distance between pixel e and the side ac. The distance th between pixel e and the plane c1c2d1d2 can be determined by the distance between pixel e and the side cd in the figure. th is equal to the distance between pixel e and the side cd. The distance tl between pixel e and the plane a2b2d2c2 can be determined based on the luminance of pixel e and the luminance value corresponding to the plane a2b2d2c2.

[0159] After determining the position of pixel e, in the embodiments of the present application, the specific method for calculating the enhancement coefficient ef of pixel e by trilinear interpolation is shown in the following formula:

[0160] ef = a1 * (1 - tw) * tl * th + b1 * tw * tl * th + c1 * (1 - tw) * tl * (1 - th) + d1 * tw * tl * (1 - th) + a2 * (1 - tw) * (1 - tl) * th + b2 * tw * (1 - tl) * th + c2 * (1 - tw) * (1 - tl) * (1 - th) + d2 * tw * (1 - tl) * (1 - th).

[0161] It should be noted that the above display resolution and region division methods are only some examples of this application and do not limit this application in any way.

[0162] The above process describes how to determine the enhancement coefficients corresponding to the first type of pixels and the second type of pixels theoretically, as well as the method of correcting pixel values according to the enhancement coefficients. In the embodiments of this application, the electronic device needs to correct the calculated enhancement coefficients according to the predicted pixel offset. The following describes the process of predicting pixel offset.

[0163] In the embodiments of this application, as Figure 2a shown, the electronic device may further include motion sensors, such as a gyroscope, an acceleration sensor, etc. The electronic device can obtain the historical data of the motion sensors before collecting the second image, and predict the pixel offset of the second image relative to the first image according to the historical data and the first image. Among them, the historical data of the motion sensors can be the motion data of the electronic device collected by the motion sensors before the camera captures the second image, such as the angular velocity collected by the gyroscope and the acceleration collected by the acceleration sensor.

[0164] The electronic device can obtain the historical motion data of the camera according to the historical data of the motion sensors. For example, the historical data of the motion sensors can be used as the historical motion data of the camera. As Figure 3 shown, the electronic device can obtain the historical motion data of the camera. According to the historical motion data, the current path of the camera can be obtained. According to the current path of the camera and the historical motion data, the pose of the camera when collecting images of multiple future frames can be predicted, and the change in the pose of the camera is converted from the world coordinate system to the change in the spatial position of the corresponding pixels in the image plane coordinate system, that is, the pixel offset of the second image relative to the first image. The specific conversion process can refer to the relevant conversion technologies from camera coordinates to image coordinates.

[0165] The pose in the embodiments of this application may refer to the pose of the camera, which is represented by M i (r x , r y , r z , t x , t y , t z ) for the motion of the i-th frame of the camera. Among them, r represents rotation, and r x , r y , r z respectively represent the angles of rotation in the x, y, and z coordinate axes directions of the world coordinate system, and t represents translation, and t x , t y , t zrespectively represent the distances of translation in the x, y, and z coordinate axes directions. Figure 6a A schematic diagram showing the pose of a camera and the pose difference according to an embodiment of the present application, Figure 6a In the shown coordinate system, the movement of the i-th frame of the camera (the pose difference of the i-th frame relative to the (i - 1)-th frame image) can be expressed as M as described above i (r x , r y , r z , t x , t y , t z ).

[0166] In an embodiment of the present application, the electronic device can obtain the angular change of the camera when collecting each frame of image relative to when collecting the previous frame of image (r x , r y , r z ) according to the angular velocity data collected by the gyroscope, and can obtain the displacement change of the camera when collecting each frame of image relative to when collecting the previous frame of image (t x , t y , t z ) according to the acceleration data collected by the acceleration sensor. According to the angular change and displacement change of the camera when collecting each frame of image relative to when collecting the previous frame of image, the movement path of the camera when shooting two frames of images can be obtained. Therefore, the electronic device can obtain the current path of the camera movement according to the movement data collected by the motion sensor.

[0167] Figure 6b A schematic diagram showing the camera movement and pose prediction according to an embodiment of the present application. Specifically, as Figure 3 and Figure 6b shown, the electronic device collects the (i - n)-th frame image through the camera, obtains the historical movement data of the camera before collecting the (i - n)-th frame image, and can obtain the current path of the camera according to the historical movement data. The current path can be as shown by the solid line part in Figure 6b . The movement of the camera for each frame relative to the previous frame can be expressed as M i (r x , r y , r z , t x , t y , t z ), where i represents the frame number, r represents rotation, and r x , r y , r z respectively represent the angles of rotation in the x, y, and z coordinate axes directions of the world coordinate system, and r x , r y , r zIt can be obtained from the angular velocity data collected by the gyroscope. t represents translation, and t x , t y , t z respectively represent the translation distances in the x, y, and z coordinate axes directions. t x , t y , t z It can be obtained from the acceleration data collected by the acceleration sensor. The electronic device can obtain the motion path of the camera when shooting two frames of images based on the angular change and displacement change of each frame of image collected by the camera relative to the previous frame of image collected. Finally, the solid line connection in Figure 6b can be used as the current path.

[0168] The electronic device can predict the pose of the camera when collecting multiple future frames of images based on the current path of the camera and the historical motion data. Specifically, a neural network model for motion trajectory prediction can be established, sample data of the camera motion can be collected and labeled, and the sample data of the camera motion can be input into the neural network model for training. The electronic device can input the historical motion data of the camera obtained in the embodiments of the present application into the trained neural network for inference, and can predict the pose of the camera when collecting several future frames of images. It should be noted that using the neural network model to predict the motion trajectory is only an example of the present application and does not limit the present application in any way. The predicted pose can be as shown by the dotted line part in Figure 6b .

[0169] It should be noted that during the prediction process, the electronic device can obtain the motion data collected by the motion sensor in real time, and use the real-time obtained motion data to predict the future pose of the camera. That is, when predicting the pose of the camera after the i - nth frame, the electronic device can use part of the motion data collected by the motion sensor after the camera collects the (i - n)th frame of image and before collecting the i - th frame of image. As shown in Figure 3 , after the electronic device collects the (i - n)th frame of image through the camera, it calculates the second enhancement coefficient based on the (i - n)th frame of image. The electronic device can also predict the pixel offset. After calculating the second enhancement coefficient, the first enhancement coefficient is obtained by adjusting the second enhancement coefficient using the pixel offset. Therefore, the historical motion data used by the electronic device during the prediction of the pixel offset can be the motion data collected by the motion sensor before the electronic device calculates the second enhancement coefficient.

[0170] Such as Figure 3As shown in the figure, the electronic device can predict the first delay frame number of the second image relative to the first image. In an embodiment of the present application, for two of a plurality of continuously acquired images: the first image and the second image, the acquisition time of the second image is after the first image. An enhancement coefficient is calculated based on the first image, and the frame to which the enhancement coefficient applies is the second image. The number of frames between the second image and the first image represents the delay frame number.

[0171] As Figure 3 shown, in a possible implementation, the electronic device can determine the first delay frame number of the second image relative to the first image according to the historical data of the statistically obtained delay frame number (the second delay frame number). For example, the electronic device can record the second delay frame number corresponding to each frame, and predict the future first delay frame number according to the average value of the recorded second delay frame numbers: DelayNum = f(delay1, delay2,... delayN), where DelayNum represents the predicted future delay frame number of the second image relative to the first image (the first delay frame number), delay1, delay2,... delayN respectively represent the second delay frame numbers of the previously recorded frames, and f represents the function for calculating the average value. For example, DelayNum = (delay1 + delay2 +... + delayN) / N, where N represents the number of the second delay frame numbers, that is, the second delay frame numbers corresponding to N frames are statistically obtained, and N is a positive integer. In an embodiment of the present application, the "first" and "second" in the first delay frame number and the second delay frame number are used to distinguish different types of delay frame numbers and do not represent an order relationship. The second delay frame number can represent the historical data of the statistically obtained delay frame number, and the first delay frame number can represent the predicted future delay frame number.

[0172] For example, as Figure 3 and Figure 6b shown, for N frames before the (i - n)-th frame, the electronic device records the timestamps of these N frames (the first images) and the timestamps of the frames (the second images) to which the enhancement coefficients calculated based on these N frames apply. The electronic device can obtain a plurality of second delay frame numbers, that is, the second delay frame numbers delay1, delay2,... delayN corresponding to the above N frames, according to the timestamps of the plurality of first images before the (i - n)-th frame and the timestamps of the second images corresponding to the plurality of first images. Calculating the average value of the second delay frame numbers can be used as the predicted first delay frame number of the second image relative to the first image.

[0173] In another possible implementation, the electronic device can directly determine the first delay frame number of the second image relative to the first image based on the recorded timestamp information. The electronic device can record the timestamp when starting to calculate the enhancement factor and the timestamp when finishing calculating the enhancement factor, and determine the delay frame number of the predicted frame relative to the current frame in combination with the timestamp information: DelayNum = Ceil((CurrentStamp – LastFrameStamp) / FrameIntervel), where CurrentStamp represents the timestamp when the enhancement factor correction is completed, LastFrameStamp represents the timestamp of the first image, FrameIntervel represents the frame interval between two frames, FrameIntervel can be obtained according to the frame rate, FrameIntervel is the reciprocal of the frame rate. For example, if the frame rate is 30fps, then the frame interval can be 1s / 30, and ceil means rounding up. For example, assume that the timestamp of the recorded first image is t1, the timestamp after correcting the enhancement factor is t2, and the frame interval is T. If (t2 - t1) / T is an integer, DelayNum = (t2 - t1) / T. If (t2 - t1) / T is not an integer, the number obtained after rounding up is DelayNum. For example, if (t2 - t1) / T is 2.3, then the delay frame number DelayNum is 3.

[0174] By continuously predicting the pose difference and delay frame number of the camera in the above embodiments and correcting the contrast of the second image according to the prediction results, the flicker of the video or preview image caused by the fluctuation of the frame delay can be alleviated. That is to say, the image processing method of the present application makes the fluctuation of the frame delay predictable to a certain extent by predicting the pixel offset, and correcting the contrast of the second image according to the prediction results can improve the display effect.

[0175] As Figure 3 shown, the electronic device can determine the pixel offset of the second image relative to the first image according to the predicted pose of the camera in the next few frames and the first delay frame number of the second image relative to the first image.

[0176] Specifically, the electronic device can determine the poses of the camera corresponding to the first delay frame number from the predicted poses of the camera in the next few frames. The electronic device can determine the pose difference between the pose of the camera when collecting the second image and the pose of the camera when collecting the first image according to the poses of the camera corresponding to the first delay frame number, and convert the pose difference of the camera from the world coordinate system to the change in the spatial position of the corresponding pixel in the image plane coordinate system, that is, the pixel offset of the second image relative to the first image. The specific conversion process can refer to the relevant conversion technology from camera coordinates to image coordinates.

[0177] For example, assume that the electronic device predicts the poses of the camera corresponding to the 5 frames of images after the first image (the poses of the camera at the moments when the 5 frames of images are captured): M i (r x ,r y ,r z ,t x ,t y ,t z ), i = 1, 2, 3, 4, 5. The first number of delayed frames determined by the electronic device is 3 frames. Therefore, the electronic device can determine the pose difference of the camera at the moment of obtaining the second image relative to the pose at the moment of obtaining the first image according to the predicted poses of the camera corresponding to the 3 frames of images after the first image. That is to say, the pose difference can be determined according to M i (r x ,r y ,r z ,t x ,t y ,t z ), i = 1, 2, 3. The pose difference of the camera is obtained, and the change in the spatial position of the corresponding pixels of the pose difference from the world coordinate system to the image plane coordinate system is calculated, which is the pixel offset of the second image relative to the first image. For example, if M1(0, 0, 0, 0.01, 0.01, 0), M2(0, 0, 0, 0, 0.01, 0), M3(0, 0, 0, -0.01, 0, 0), then from the start of shooting the first image to shooting the second image, the pose difference of the camera can be expressed as (0, 0, 0, 0, 0.02, 0), that is, the distance the camera moves in the Y direction is 0.02, and the distances moved and rotation angles in other directions are all 0. The unit of distance can be cm. Multiplying the above pose difference of the camera by the rotation matrix and the translation matrix can obtain the coordinate change of the camera in the camera coordinate system. Performing imaging projection (converting to the image plane coordinate system) on the coordinate change of the camera in the camera coordinate system can obtain the pixel offset in the image plane coordinate system, which is the pixel offset of the second image relative to the first image. The specific conversion process can refer to the related conversion technologies from camera coordinates to image coordinates.

[0178] Figure 7 FIG. shows a schematic diagram of the pixel offset of the second image relative to the first image according to an embodiment of the present application. As Figure 7 shown, the solid rectangle part can represent the first image and the grid of the divided areas on the first image. Four pixels a, b, c, and d all have corresponding enhancement coefficients. The dashed rectangle can represent the second image and the grid of the divided areas on the second image. Assume that when the user holds the camera to take pictures, it is found that part of both flowers is not in the lens. For example Figure 7Part of the lower part of the flower on the left is not in the camera's view, and part of the right side of the flower on the right is not in the camera's view. Therefore, the user pans the camera to the right and down to make both flowers fully within the camera's view, as Figure 7 shown by the dashed rectangular box in Figure 7 . When the camera captures the second image, it pans to the right and down relative to when it captures the first image, resulting in a pixel offset of the second image relative to the first image in the image plane coordinates. Therefore, the relative position of the coordinates of the same pixel in the second image and the first image with respect to the coordinate origin has changed. For example, for pixel a in the first image, the original position coordinates (relative to the coordinate origin O) were (w2, h1). After panning to the right by wd in the W direction and down by hd in the H direction, in the second image, the position coordinates of pixel a (relative to the coordinate origin O1) can be (w2 - wd, h1 + hd).

[0179] As Figure 3 shown, the electronic device can correct the second enhancement coefficient based on the pixel offset of the second image relative to the first image to obtain the first enhancement coefficient.

[0180] Specifically, the electronic device can adjust the relationship between the position coordinates of the pixels in the first lookup table and the second lookup table based on the change in the spatial position of the pixels (pixel offset), and / or the corresponding relationship between the brightness coordinates of the pixels in the second lookup table and the second enhancement coefficient. That is to say, the process of correcting the enhancement coefficient in this application can be to adjust the corresponding relationship between the pixels and the enhancement coefficients corresponding to the pixels, such as adjusting the corresponding relationship between the position coordinates, brightness coordinates, and enhancement coefficients. In fact, the value of the enhancement coefficient is not adjusted. The image processing method provided by the embodiments of this application can adjust the relationship between the position coordinates of the pixels in the first lookup table and the second lookup table based on the change in the position of the pixels in the second image relative to the first image, and / or the corresponding relationship between the brightness coordinates of the pixels in the second lookup table and the enhancement coefficient.

[0181] The electronic device captures the second image (the i-th frame image) through the camera and corrects the contrast of the second image according to the first enhancement coefficient.

[0182] Taking the first lookup table and the second lookup table in the above example as an example, assume that the right and downward movement of the camera causes the position of the pixels in the second image relative to the first image to pan to the right by wd in the W direction and down by hd in the H direction. The electronic device can adjust the first lookup table, and the adjusted first lookup table is shown in Table 3:

[0183] Table 3

[0184] w1 - wd w2 - wd …… wn - wd h1 + hd Table 11 Table 12 Table 1n h2 + hd Table 21 Table 22 Table 2n …… hm + hd Table m1 Table m2 …… Table mn

[0185] In this way, when the electronic device queries the enhancement coefficient corresponding to the pixel in the second image according to the adjusted first query table and the second query table, it can obtain the enhancement coefficient that matches the pixel, which can minimize the spatial deviation between the enhancement coefficient and the frame being processed, thereby reducing the inter-frame difference and improving the enhancement effect.

[0186] Taking pixel a as an example, the position coordinates of pixel a in the second image are (w2 - wd, h1 + hd). By querying the adjusted first query table (Table 3), the second query table corresponding to pixel a can be obtained as Table 12. According to Table 12 and the brightness of pixel a, the first enhancement coefficient corresponding to pixel a can be determined. Similarly, according to the position coordinates (w2, h1) of pixel a in the first image, by querying the first query table before adjustment, the second query table corresponding to pixel a can be obtained as Table 12. According to Table 12 and the brightness of pixel a, the first enhancement coefficient corresponding to pixel a can be determined. By comparison, it can be seen that the query results are the same. By correcting the enhancement coefficient according to the pixel offset of the second image relative to the first image in the electronic device, the spatial deviation between the enhancement coefficient and the image of the frame being processed can be minimized, thereby reducing the inter-frame difference and improving the enhancement effect.

[0187] When the electronic device displays the image collected by the camera, it can use the corrected second image to replace the second image collected by the camera for display, that is, the electronic device displays the corrected second image.

[0188] It should be noted that the above process of correcting the enhancement coefficient according to the pixel offset is only an example of this application, and this application is not limited thereto.

[0189] The electronic device can correct the image information of the first image according to the pixel offset of the second image relative to the first image, calculate the enhancement coefficient according to the corrected first image, and correct the image contrast of the second image according to the enhancement coefficient. Figure 8 A frame-by-frame timing diagram of the image processing method according to an embodiment of this application is shown.

[0190] Combined Figure 2a and Figure 8 as shown, the image processing method of this example is described. As Figure 2a shown, the electronic device according to an embodiment of this application may further include a motion sensor, and the electronic device can obtain the historical data of the motion sensor before the camera collects the second image.

[0191] As Figure 8 shown, the electronic device obtains the historical motion data of the camera through the motion sensor, and predicts the pose of the camera in the next few frames according to the historical motion data of the camera.

[0192] Specifically, the electronic device can obtain the current path of the camera based on the historical motion data of the camera, and predict the poses of the camera in the next few frames according to the current path of the camera and the historical motion data. The specific process can be referred to above and will not be elaborated here.

[0193] In this embodiment, the electronic device can predict the poses of the camera when capturing the next few frames after capturing the first image based on the historical data of the motion sensor before the camera captures the first image.

[0194] The electronic device can predict the first delay frame number of the second image relative to the first image.

[0195] As Figure 8 shown, the electronic device can determine the average delay frame number as the first delay frame number of the second image relative to the first image according to the historical data of the delay frame number (the second delay frame number). For example, the electronic device can record the second delay frame number corresponding to each frame, and predict the future first delay frame number according to the average value of the recorded second delay frame numbers ( Figure 8 the average delay frame number shown). The specific process can be referred to the above introduction and will not be elaborated here. Since the enhancement coefficient has not started to be calculated, in this example, the average delay frame number can be used as the first delay frame number, and the historical data of the motion sensor before capturing the first image can be used to predict the pose of the camera in this example.

[0196] The electronic device can determine the pixel offset of the second image relative to the first image according to the predicted poses of the camera in the next few frames and the first delay frame number of the second image relative to the first image. As Figure 8 shown, the electronic device can predict the pixel offset of the i-th frame relative to the (i - n)-th frame. The specific process can be referred to the above introduction and will not be elaborated here.

[0197] The electronic device can correct the image information of the first image according to the pixel offset of the second image relative to the first image to obtain the third image. Specifically, the spatial position of the pixels of the first image can be corrected. As Figure 8 shown, the electronic device can adjust the image information of the (i - n)-th frame image according to the pixel offset.

[0198] As Figure 7 shown, assume that the rightward and downward movement of the camera causes the position of the pixels in the second image relative to the first image to be translated rightward by wd in the W direction and downward by hd in the H direction. Assume that the coordinate of pixel a in the first image before adjustment is (w2, h1). According to the pixel offset, the coordinate of pixel a in the third image is (w2 - wd, h1 + hd). According to the above process, adjusting the pixels in the first image that are the same as those in the second image can obtain the third image.

[0199] In a possible implementation, the electronic device may calculate a transformation matrix based on the pose difference (pixel offset), and then apply the transformation matrix to the first image. For example, the sum and / or difference of the first image and the transformation matrix may be obtained to get the third image.

[0200] In the embodiments of the present application, the transformation matrix H = Fun(rot_predict - rot_current), where rot_predict represents the predicted pose of the camera when collecting the second image, and rot_current represents the pose of the camera when collecting the first image. Fun represents a function for calculating the transformation matrix based on the pose difference. As described above, multiplying the pose difference by the rotation matrix and the translation matrix can obtain the coordinate change of the camera in the camera coordinate system. Performing imaging projection (converting to the image plane coordinate system) on the coordinate change of the camera in the camera coordinate system can obtain the pixel offset in the image plane coordinate system, and thus the transformation matrix H can be obtained.

[0201] The third image I’ = H(I in )), where I in represents the first image.

[0202] As Figure 8 shown, the electronic device may obtain statistical information based on the image information of the third image, and calculate a first enhancement coefficient according to the statistical information of the third image.

[0203] The electronic device collects the second image (the i-th frame image) through the camera, and may correct the contrast of the second image according to the first enhancement coefficient.

[0204] The enhancement coefficient calculated based on the image information of the corrected first image is matched with the pixels of the predicted second image. Correcting the contrast of the collected second image can minimize the spatial deviation between the enhancement coefficient and the acting frame, thereby reducing the inter-frame difference and improving the enhancement effect.

[0205] The electronic device may use the second image with corrected contrast to replace the second image collected by the camera for display.

[0206] Figure 9a Fig. shows an inter-frame timing diagram of an image processing method according to an embodiment of the present application. In this example, the electronic device may predict the pixel offset of the second image relative to the first image after the first image in multiple frames based on the information of the historical image before the second image and the first image.

[0207] In the scenario of this example, the camera can be the camera on a mobile phone. The user holds the mobile phone to preview before shooting a video or taking a photo. The hand holding the mobile phone can be stationary, and there can be a moving target within the camera's viewfinder. The camera can also be the camera installed on a drone. The drone hovers for shooting, and there can be a moving target within the camera's viewfinder. Or, the camera can also be a vehicle-mounted camera (dash cam). The car is parked in a fixed position, and the camera shoots the surrounding area of the vehicle, and there can be a moving target in the captured image, and so on. This application does not limit the specific application scenarios.

[0208] Combined with Figure 2a and Figure 9a as shown, the image processing method of this example will be described.

[0209] As Figure 9a shown, the electronic device can predict the first delay frame number of the second image relative to the first image. The specific process can refer to the above introduction and will not be elaborated here.

[0210] As Figure 9a shown, the electronic device can predict the pixel offset of several future frames relative to the first image based on the image information of the historical images before the second image and the first image.

[0211] Specifically, a neural network model can still be established. The neural network model can include a moving target recognition module and a motion trajectory prediction module. Sample data including moving targets are collected and labeled. The sample data including moving targets are input into the neural network to train the neural network model. Specifically, the moving target recognition module can extract the optical flow features of the input sample data, identify the moving target according to the optical flow features, determine the motion trajectory of the moving target based on the moving targets in multiple frames of images. The motion trajectory prediction module predicts the future motion trajectory of the moving target according to the identified motion trajectory, and calculates the loss function based on the prediction result and the actual result. If the loss function does not meet the requirements, reverse inference is performed to adjust the parameters of the neural network model and continue training.

[0212] Input the historical images of the embodiments of this application into the trained neural network model for forward inference, the moving targets in the historical images can be identified, and the motion trajectories of the moving targets can be predicted. According to the prediction result, the pixel offset of several future frames relative to the first image can be obtained.

[0213] As Figure 9a shown, the electronic device can combine the predicted pixel offset of several future frames relative to the first image and the first delay frame number to obtain the pixel offset of the second image relative to the first image.

[0214] Figure 9bSchematic diagram showing the prediction of the target motion trajectory according to an embodiment of the present application. As Figure 9b shown, the image on the left represents the historical image before the first image, and the historical image includes a moving target. As Figure 9b shown, each frame of the image includes a black background, and the black background includes the white number "5", and the position of "5" in each frame of the image is different and is gradually moving. According to the historical image, the motion trajectory of the moving target in the next 5 frames of images after the first image can be predicted. Assuming that the first delay number of frames of the second image relative to the first image is 5 frames, the pixel offset of the second image relative to the first image can be determined according to the predicted 5th frame image.

[0215] In another possible implementation, the electronic device can also perform the conversion from the image plane coordinate system to the camera coordinate system and the world coordinate system according to the image information of the historical image, perform 3D modeling, and then, the historical pose of the camera when collecting the historical image can be determined according to the obtained 3D model. According to the historical pose, the pose of the camera when collecting the next few frames of images can be predicted. The pose difference of the pose of the camera when collecting the second image relative to the pose when collecting the first image is determined according to the predicted pose of the camera when collecting multiple frames of images after the first image and the first delay number of frames. According to the pose difference, the pixel offset of the second image relative to the first image can be determined.

[0216] The electronic device can also combine the above two methods to predict the pixel offset of the second image relative to the first image according to the information of the historical image before the second image and the first image.

[0217] It should be noted that Figure 9a the process of obtaining the enhancement coefficient shown is an example of first adjusting the first image according to the pixel offset to obtain the third image, and then calculating the first enhancement coefficient. If the example of calculating the first enhancement coefficient in Figure 3 is adopted, the pixel offset of the next few frames of images relative to the first image can be predicted according to the historical image before the second image.

[0218] Figure 9c Schematic diagram showing the pixel offset of the second image relative to the first image according to an embodiment of the present application. As Figure 9c shown, the solid rectangle part can represent the first image (the (i - n)th frame image) and the grid of the divided area on the first image, and the dashed rectangle can represent the second image (the ith frame image) and the grid of the divided area on the second image. Assuming that during the shooting process of the camera, the lens does not move and a car drives past the lens, the process of the car's movement can be predicted according to the image processing method of the present application, the position of the car in the second image can be obtained, and the pixel offset of the second image relative to the first image can be determined according to the predicted motion trajectory. As Figure 9cThe pixel offset of the pixels corresponding to the vehicle in the middle. The static targets and background parts in the figure may not have pixel offsets. If the camera lens moves or rotates during the shooting process, pixel offsets may also exist in the static targets and background parts.

[0219] To determine the pixel offset of the second image relative to the first image, the electronic device can calculate the second enhancement coefficient based on the first image, correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient; alternatively, the electronic device can also correct the first image according to the pixel offset to obtain the third image, and calculate the first enhancement coefficient based on the third image. This example does not limit the process of obtaining the first enhancement coefficient based on the pixel offset and the first image, and one of the methods will be used to introduce this example.

[0220] As Figure 9a shown, the electronic device can adjust the image information of the first image according to the predicted pixel offset of the second image relative to the first image to obtain the third image.

[0221] Specifically, according to Figure 9b the example shown, the electronic device can adjust the position coordinates of the corresponding pixels in the first image according to the pixel offset, so that the moving target in the adjusted third image is located at the predicted position.

[0222] As Figure 9a shown, the electronic device can calculate the first enhancement coefficient according to the image information of the third image. Taking Figure 4 and Figure 5a the application examples shown as an example, the electronic device can partition the pixels on the third image, statistically obtain the statistical information of each region of the third image, and calculate according to the statistical information of each region to obtain the first enhancement coefficient of each region (the first type of pixels). The electronic device can obtain the mapping relationship between the first type of pixels and the corresponding first enhancement coefficients.

[0223] The electronic device can establish the first query table and the second query table according to the mapping relationship between the first type of pixels and the corresponding first enhancement coefficients.

[0224] The electronic device acquires the second image (the i-th frame image) through the camera. The electronic device can correct the contrast of the second image according to the calculated first enhancement coefficient. The electronic device can use the second image with the corrected contrast to replace the second image acquired by the camera for display.

[0225] According to the image processing method of the above example of the present application, the pixel offset of the future image relative to the current image is predicted through the image information of the historical image, the first enhancement coefficient is obtained according to the current image and the pixel offset, and the image contrast of the future image is enhanced according to the obtained first enhancement coefficient, which can minimize the spatial deviation between the first enhancement coefficient and the frame of action, thereby reducing the inter-frame difference and improving the enhancement effect.

[0226] Figure 10 The inter-frame timing diagram showing the image processing method according to an embodiment of the present application is shown. In this example, the electronic device can predict the pixel offset of the second image relative to the first image according to the historical data of the motion sensor and the information of the historical image before collecting the second image, and the first image.

[0227] Combined with Figure 2a and Figure 10 shown, the image processing method of this example will be described. As Figure 2a shown, the electronic device according to the embodiment of the present application may further include a motion sensor, and the electronic device can obtain the historical data of the motion sensor before the camera collects the second image.

[0228] As Figure 10 shown, in this example, the electronic device can predict the first delay number of frames of the second image relative to the first image. The specific process can be referred to the above, and will not be repeated here.

[0229] The electronic device can predict the pixel offset of the second image relative to the first image according to the image information of the historical image before the second image, the historical data of the motion sensor, and the above first delay number of frames.

[0230] Specifically, the processor predicts the poses of the future several frames according to the historical data of the motion sensor, and the first pixel offset of the future several frames relative to the first image can be obtained according to the pose difference between the poses of the camera when collecting the future several frames and the pose when collecting the first image; the processor predicts the second pixel offset of the future several frames relative to the first image according to the historical image, and the pixel offset of the future several frames relative to the first image can be obtained according to the first pixel offset and the second pixel offset.

[0231] The electronic device predicts the second pixel offset of the future several frames of images relative to the first image based on historical images. Specifically, the process may include the following: The electronic device can perform the conversion from the image plane coordinate system to the camera coordinate system and the world coordinate system according to the image information of the historical images, and perform 3D modeling. Then, the historical pose can be determined according to the obtained 3D model. According to the historical pose, the pose when the camera captures the future several frames of images can be predicted. The second pixel offset of the future several frames of images relative to the first image can be obtained according to the pose difference between the pose when the camera captures the future several frames of images and the pose when the first image is captured.

[0232] The electronic device combines the predicted pixel offset of the future several frames of images relative to the first image and the first delay number of frames to obtain the pixel offset of the second image relative to the first image.

[0233] Such as Figure 10 As shown, the electronic device can correct the image information of the first image according to the predicted pixel offset of the second image relative to the first image to obtain the third image; calculate the first enhancement coefficient according to the image information of the third image.

[0234] The electronic device captures the second image (the i-th frame image) through the camera, and corrects the contrast of the second image according to the first enhancement coefficient.

[0235] The electronic device can use the second image with corrected contrast to replace the second image captured by the camera for display.

[0236] It should be noted that in this example, the electronic device can also calculate the second enhancement coefficient according to the first image, correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient. The specific process of correcting the enhancement coefficient according to the pixel offset and the spatial difference of the acting frames is not limited in this application.

[0237] It should be noted that the prediction process of the pixel offset and the process of adjusting the spatial position difference according to the pixel offset are not specifically limited in this application. For example, in the embodiments of this application, the electronic device can also obtain the third pixel offset of the second image relative to the first image according to the first pixel offset and the first delay number of frames, and obtain the fourth pixel offset of the second image relative to the first image according to the second pixel offset and the first delay number of frames. The electronic device can correct the image information of the first image according to the fourth pixel offset, calculate the enhancement coefficient according to the corrected first image, and the electronic device can correct the enhancement coefficient according to the third pixel offset, and then correct the contrast of the second image according to the corrected enhancement coefficient. That is, first correct the first image according to the local pixel offset, calculate the enhancement coefficient according to the corrected first image, correct the enhancement coefficient according to the global pixel offset, and correct the contrast of the second image according to the corrected enhancement coefficient.

[0238] The image processing methods according to the above embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the action frame, thereby reducing the inter-frame difference and improving the enhancement effect.

[0239] Figure 11a The inter-frame timing diagram showing the image processing method according to an embodiment of the present application. In the embodiment of the present application, as described in the above embodiment, according to the first image, predicting the pixel offset of the second image relative to the first image after the first image among multiple frames of images may include: predicting the pose difference of the pose of the camera when acquiring the second image relative to the pose when acquiring the first image (the predicted pose difference of the camera); obtaining the pixel offset of the second image relative to the first image according to the pose difference.

[0240] In a possible implementation, the electronic device may adjust the subsequent process of predicting the pixel offset according to the error between the predicted pose difference of the camera and the actual pose difference. Wherein, the actual pose difference is the difference between the pose of the camera when actually acquiring the second image and the pose when acquiring the first image. Specifically, the electronic device is further configured to, when the historical pose prediction accuracy before the second image does not meet the accuracy condition, adjust the subsequent process of predicting the pose of the camera according to the predicted pose difference of the camera, wherein the historical pose prediction accuracy is obtained according to the predicted pose difference of the camera and the corresponding actual pose difference before acquiring the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet.

[0241] That is to say, in the embodiment of the present application, before the electronic device determines the pixel offset using the predicted pose difference of the camera, it may determine whether the historical pose prediction accuracy meets the accuracy condition.

[0242] If the historical pose prediction accuracy meets the accuracy condition, the electronic device may determine the pixel offset of the second relative to the first image according to the predicted pose difference of the camera, and correct the first image or the enhancement coefficient according to the pixel offset.

[0243] If the historical pose prediction accuracy does not meet the accuracy condition, the electronic device may adjust the subsequent process of predicting the poses of the next few frames of images. Specifically, the electronic device may include a prediction module for predicting poses. The prediction module may use an AI model or other models to predict poses. The electronic device may adjust the parameters of the prediction module according to the error between the predicted pose difference and the actual pose difference, such as adjusting the parameters of the AI model, so as to achieve the adjustment of the subsequent process of predicting poses.

[0244] As Figure 11a shown, the image processing method of this example may include the following process:

[0245] The electronic device can predict the first number of delayed frames of the second image relative to the first image. For the specific process, please refer to the above, and details will not be repeated here.

[0246] The electronic device predicts the pose difference of the images in the next few frames relative to the first image based on the image information of the historical images before the second image and / or the historical data of the motion sensor.

[0247] Combining the predicted pose difference of the images in the next few frames relative to the first image and the first number of delayed frames, the electronic device can obtain the pose difference of the second image relative to the first image.

[0248] As Figure 11a shown, the electronic device can determine whether the historical pose prediction accuracy meets the accuracy condition. If it meets, the pixel offset of the second image relative to the first image can be obtained according to the predicted pose difference of the second image relative to the first image, and the process of correcting the contrast of the subsequent corrected image can be performed. For example, the electronic device can correct the image information of the first image according to the predicted pixel offset of the second image relative to the first image to obtain a third image; calculate a first enhancement coefficient according to the image information of the third image; and correct the contrast of the second image according to the first enhancement coefficient. If it does not meet, the electronic device can obtain the true pose difference of the second image relative to the first image and adjust the process of the predicted pose according to the error between the predicted pose difference and the true pose difference.

[0249] In a possible implementation, the electronic device can statistically calculate the historical pose prediction accuracy in the form of a sliding window. Figure 11b A schematic diagram showing the determination of the historical pose prediction accuracy according to an embodiment of the present application is as Figure 11b shown. The electronic device can statistically calculate and calculate the average value of the pose prediction accuracy as the historical pose prediction accuracy according to the pose prediction accuracy corresponding to the images from the (i - n - k)-th frame to the (i - n - l)-th frame, where k and l are both positive integers, and k is greater than l. For example, assuming l is 1 and k is 5, then the length of the sliding window is 4. The pose prediction accuracy corresponding to the (i - n - 5)-th frame image is pdiff(i - n - 5), the pose prediction accuracy corresponding to the (i - n - 4)-th frame image is pdiff(i - n - 4), the pose prediction accuracy corresponding to the (i - n - 3)-th frame image is pdiff(i - n - 3), the pose prediction accuracy corresponding to the (i - n - 2)-th frame image is pdiff(i - n - 2), and the pose prediction accuracy corresponding to the (i - n - 1)-th frame image is pdiff(i - n - 1). Therefore, the electronic device can obtain the historical pose prediction accuracy as pdiff = (pdiff(i - n - 5) + pdiff(i - n - 4) + pdiff(i - n - 3) + pdiff(i - n - 2) + pdiff(i - n - 1)) / 4.

[0250] Assume that the precision condition is that pdiff is greater than pdiff(ref), where pdiff(ref) is the pose prediction precision threshold. If the historical pose prediction precision is greater than pdiff(ref), the electronic device can determine the pixel offset according to the predicted pose difference and correct the process of enhancing the image contrast according to the pixel offset; if the historical pose prediction precision is not greater than pdiff(ref), as described above, the electronic device can adjust the process of predicting the pose of the future image according to the error between the predicted pose difference and the true pose difference, as Figure 11a shown.

[0251] It should be noted that Figure 11a the sequence relationship between the step of determining whether the historical pose prediction precision meets the precision condition and other steps in [] is only an example of this application and does not limit this application in any way. For example, in the example of this application, the electronic device can continuously determine whether the historical pose prediction precision meets the precision condition. For example, it can determine whether the historical pose prediction precision meets the precision condition at a certain time interval, regardless of which step the subsequent prediction reaches. If it is determined that the historical pose prediction precision does not meet the precision condition, the process of adjusting the prediction is executed until it is determined that the historical pose prediction precision meets the precision condition. Then, it can execute determining the pixel offset according to the pose difference, and according to the pixel offset, the image information of the first image can be adjusted to obtain the third image. The first enhancement coefficient is calculated according to the image information of the third image, and the image contrast of the second image is enhanced according to the first enhancement coefficient. After that, it still continuously determines whether the historical pose prediction precision meets the precision condition. If it does not meet the condition, the prediction process is adjusted according to the currently obtained pose difference and the corresponding true pose difference. If it meets the condition, the pixel offset is determined according to the currently obtained pose difference and the subsequent process is executed.

[0252] It should be noted that in this example, the electronic device can calculate the second enhancement coefficient according to the first image, correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient; or, the electronic device can also correct the first image according to the pixel offset to obtain the third image, and calculate the first enhancement coefficient according to the third image. The electronic device acquires the second image (the i-th frame image) through the camera and corrects the contrast of the second image according to the first enhancement coefficient.

[0253] In the above embodiments of this application, the electronic device displays the corrected second image instead of the second image acquired by the camera. By adjusting the prediction process through the feedback of the prediction result, the prediction accuracy can be further improved, so that the spatial deviation between the enhancement coefficient and the effective frame is minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0254] In a possible implementation, the electronic device can select the information based on which to predict the poses of future several frames of images according to the motion scenario. The electronic device may be in a motion scenario with a relatively large motion amplitude, or may be in a relatively stationary motion scenario with a relatively small motion amplitude. For example, when the user holds the electronic device and takes pictures while walking, or when the user holds the electronic device and takes pictures in a moving vehicle, the motion amplitude of the electronic device is relatively large in these scenarios. If the electronic device is assumed to be on a bracket, or the user holds the electronic device to take pictures, but the user does not move and the holding method is relatively stable, the motion amplitude of the electronic device is relatively small in these scenarios.

[0255] In the embodiments of the present application, as Figure 2a shown, the electronic device may further include a motion sensor. The electronic device can obtain the historical data of the motion sensor. The historical data of the motion sensor may refer to the motion data of the electronic device collected by the motion sensor before the camera collects the second image. The electronic device can determine the motion scenario according to the historical data, and predict the pixel offset of the second image relative to the first image according to the historical motion information corresponding to the motion scenario; wherein, the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is the information of the historical image.

[0256] Figure 12 The frame - by - frame timing diagram showing the image processing method according to an embodiment of the present application is as follows. As Figure 12 shown, the image processing method of this embodiment may include the following processes:

[0257] The electronic device can predict the first delay number of frames of the second image relative to the first image.

[0258] The electronic device determines the motion scenario where the electronic device is located through the data sensed by the motion sensor. If the electronic device is in Scenario 1 (the first scenario), the electronic device can predict the pose difference of the camera when collecting future several frames of images relative to when collecting the first image according to the historical data of the motion sensor and the first image. If the electronic device is in Scenario 2 (the second scenario), the electronic device can predict the pose difference of the camera when collecting future several frames of images relative to when collecting the first image according to the image information of the historical image before the second image and the first image.

[0259] The process after predicting the pose difference can be the same as the processing process in the previous embodiments, and can be combined with the process of the previous embodiments, which will not be elaborated here.

[0260] In the embodiments of the present application, the electronic device may determine the current motion scenario of the electronic device according to the historical data of the motion sensor before the second image, or rather, the electronic device may determine the motion scenario of the electronic device according to the data of the motion sensor obtained in real time. The present application does not limit this. The motion sensor may include an accelerometer, a gyroscope, etc. as described above. According to the data of the motion sensor obtained, it can be determined whether the electronic device is in a first scenario with a relatively large motion amplitude or in a second motion scenario with a relatively small motion amplitude. Among them, the motion amplitude can be measured according to parameters such as the speed of the movement or rotation of the electronic device. The present application does not limit the criteria for specifically determining the motion amplitude size, and it can be set according to actual application requirements. Dividing the motion scenario into the first scenario and the second scenario in the embodiments of the present application is only an example of the present application. The present application is not limited thereto, and multiple different scenarios can also be set, and the data corresponding to each scenario is different. For example, a third scenario can also be set, and the historical motion information corresponding to the third scenario may include the historical data of the motion sensor and the information of the historical image.

[0261] In this embodiment, after predicting the pose difference, the electronic device may further determine whether the historical pose prediction accuracy meets the accuracy condition. If the accuracy condition is met, the electronic device may combine the predicted pose difference and the first delay number of frames to obtain the pixel offset of the second image relative to the first image and perform the subsequent process of correcting the contrast. For example, according to the predicted pixel offset of the second image relative to the first image, the image information of the first image can be corrected to obtain a third image; the first enhancement coefficient is calculated according to the image information of the third image; the electronic device corrects the contrast of the second image according to the first enhancement coefficient. When the electronic device displays multiple frames of images collected by the camera, the second image with the corrected contrast is used to replace the second image collected by the camera for display.

[0262] If the accuracy condition is not met, the electronic device may obtain the actual pose difference and adjust the subsequent pose prediction process according to the error between the predicted pose difference and the actual pose difference. The specific process can be referred to above and will not be elaborated here. It should be noted that in this embodiment, the electronic device may also not determine whether the historical pose prediction accuracy meets the accuracy condition and directly execute the subsequent process according to the predicted pose difference.

[0263] By pre-determining the motion scenario and selecting appropriate historical motion information according to the motion scenario to predict the poses of the next few frames of images, the prediction result can be made more accurate, so that the enhancement coefficient and the spatial deviation of the action frames are minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0264] It should be noted that the methods for predicting the first delay frame number, predicting the poses of future several frames of images, the process of adjusting and correcting the image contrast according to the pixel offset, and different methods such as the determination of the motion scene and the determination of the pose prediction accuracy in the above examples of the present application can be combined with each other, and are not limited to the specific processes of the embodiments.

[0265] Based on the above embodiments of the present application, the present application provides an image processing method. Figure 13 The flowchart showing the image processing method according to an embodiment of the present application is Figure 13 The method shown can be applied to Figure 2a and Figure 2a the electronic devices shown, such as Figure 13 as shown, the method may include:

[0266] Step S110, acquiring a first image through a camera, where the first image is one frame of a plurality of consecutive images acquired by the camera.

[0267] Step S111, predicting, according to the first image, the pixel offset of a second image after the first image in the plurality of images relative to the first image, where the pixel offset is the spatial position change of the pixels of the second image relative to the first image.

[0268] Step S112, obtaining a first enhancement coefficient according to the first image and the pixel offset.

[0269] Step S113, acquiring the second image through the camera;

[0270] Step S114, correcting the contrast of the second image according to the first enhancement coefficient.

[0271] Step S115, displaying the corrected second image.

[0272] In the image processing method provided by the present application, during the process of continuously acquiring images through a camera by an electronic device, the pixel offset of a subsequent second image relative to a previously acquired first image is predicted, the enhancement coefficient is corrected according to the pixel offset of the second image relative to the first image, and the image contrast of the second image is corrected according to the corrected enhancement coefficient. By correcting the enhancement coefficient according to the spatial position change of the pixels, the spatial deviation between the enhancement coefficient and the image of the frame being processed can be minimized, thereby reducing the inter-frame difference. Therefore, by correcting the contrast of the second image with the corrected enhancement coefficient and replacing the second image with the second image with the corrected contrast for display, the display effect can be improved.

[0273] For step S111, in one example, the electronic device can obtain the historical data of the motion sensor, and predict the pixel offset of the second image relative to the first image based on the historical data of the motion sensor and the first image. Specifically, the electronic device can predict the pose when the camera captures multiple frames of images after capturing the first image according to the historical data of the motion sensor, predict the first delay number of frames of the second image relative to the first image, determine the pose difference between the pose when the camera captures the second image and the pose when the first image is captured according to the predicted pose when the camera captures multiple frames of images after capturing the first image and the first delay number of frames, and determine the pixel offset of the second image relative to the first image according to the pose difference. Wherein, the historical data of the motion sensor is the data before the second image is captured. For the specific process, reference can be made to Figure 3 the example and introduction shown

[0274] For step S111, in another example, the electronic device can also predict the pixel offset of the second image relative to the first image based on the information of the historical images before the second image and the first image. In a possible implementation manner, the electronic device can predict the first delay number of frames of the second image relative to the first image, predict the pose when the camera captures multiple frames of images after capturing the first image according to the information of the historical images, determine the pose difference between the pose when the camera captures the second image and the pose when the first image is captured according to the predicted pose when the camera captures multiple frames of images after capturing the first image and the first delay number of frames, and determine the pixel offset of the second image relative to the first image according to the pose difference. The specific process of predicting the pose when the camera captures multiple frames of images after capturing the first image according to the information of the historical images can be that the electronic device can perform the conversion from the image plane coordinate system to the camera coordinate system and the world coordinate system according to the image information of the historical images, perform 3D modeling, and then, the historical pose when the camera captures the historical images can be determined according to the obtained 3D model, and the pose when the camera captures the next few frames of images can be predicted according to the historical pose. For the specific process, reference can be made to Figure 9a and Figure 10 the example and introduction shown

[0275] For step S111, in another example, the electronic device can also predict the pixel offset of the second image relative to the first image based on the historical data of the motion sensor and the information of the historical images before the second image. For the specific process, reference can be made to Figure 10 the example and introduction shown

[0276] For step S111, in another example, the electronic device may also determine the motion scenario according to the historical data of the motion sensor before the second image; the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is the information of the historical image; the electronic device may predict the pixel offset of the second image relative to the first image according to the historical motion information corresponding to the motion scenario and the first image. For the specific process, reference may be made to Figure 12 the examples and introductions shown.

[0277] By pre-judging the motion scenario and selecting appropriate historical motion information according to the motion scenario to predict the poses of the next few frames of images, the prediction result can be made more accurate, so that the enhancement coefficient and the spatial deviation of the action frame can be minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0278] Among them, predicting the pixel offset of the second image after the first image in the multiple frames of images relative to the first image may include predicting the pose difference of the pose of the camera when obtaining the second image relative to the pose when obtaining the first image; according to the pose difference, obtaining the pixel offset of the second image relative to the first image. For the specific process, reference may be made to the introduction in the specific embodiment part above.

[0279] In the embodiment of the present application, the image processing method may further include: if the historical pose prediction accuracy before the second image does not meet the accuracy condition, then adjusting the pose of the predicted image after according to the pose difference, where the historical pose prediction accuracy is obtained according to the predicted pose difference and the corresponding actual pose difference before the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet. For the specific process, reference may be made to Figure 11a or Figure 12 the examples and introductions shown.

[0280] By adjusting the prediction process through the feedback of the prediction result, the prediction accuracy can be further improved, so that the enhancement coefficient and the spatial deviation of the action frame can be minimized as much as possible, thereby reducing the inter-frame difference and improving the enhancement effect.

[0281] For step S112, in one example, step S112 may include calculating a second enhancement coefficient according to the first image; correcting the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient. For the specific process, reference may be made to Figure 3 the examples and introductions shown.

[0282] For step S112, in another example, step S112 may include correcting the first image according to the pixel offset to obtain a third image; calculating a first enhancement coefficient according to the third image. For the specific process, reference may be made to Figure 8 the example and introduction shown.

[0283] For steps S113 and S114, reference may be made to the above Figure 3 、 Figure 8 、 Figure 9a 、 Figure 10 、 Figure 11a and Figure 12 the process in which some electronic devices collect the i-th frame of image through a camera, correct the i-th frame of image according to the first enhancement coefficient, and display the corrected i-th frame of image.

[0284] The image processing method provided by the embodiments of the present application can be implemented in a software manner, that is, implemented at the application layer. For example, the functions corresponding to the image processing method of the present application can be implemented in a camera application. As Figure 2b shown, the image processing method provided by the embodiments of the present application can be implemented in the system library. For example, the functions of the image processing method of the present application can be implemented in the media library of the system library.

[0285] In a possible implementation manner, when the electronic device opens the camera application, the processor runs the camera application to control the camera to continuously collect multiple frames of images. For example, the multiple frames of images collected by the camera are transmitted to the system library after being processed by the ISP. For the first image in the multiple frames of images, the system library predicts the pixel offset of the second image relative to the first image according to the information of the historical images before the first image, obtains a first enhancement coefficient according to the first image and the pixel offset, and stores the first enhancement coefficient in Figure 2a the internal memory 221 shown. The process of correcting the contrast of the second image according to the first enhancement coefficient can be implemented in a hardware manner. The function of correcting the contrast of the image can be integrated on the processor (such as the ISP) of the electronic device. The ISP can query the first enhancement coefficient stored in the internal memory, correct the contrast of the second image, and return the second image with the corrected contrast to the camera application. When the camera application displays multiple frames of images, it replaces the second image with the second image with the corrected contrast for display.

[0286] In another possible implementation, when the electronic device turns on the camera application, the processor runs the camera application and controls the camera to continuously capture multiple frames of images. For example, the multiple frames of images captured by the camera are transmitted to the system library after being processed by the ISP, and the processor also obtains the historical data of the motion sensor and transmits it to the system library. For the first image among the multiple frames of images, the system library predicts the pixel offset of the second image relative to the first image according to the historical data of the motion sensor, obtains a first enhancement coefficient based on the first image and the pixel offset, and stores the first enhancement coefficient in Figure 2a the internal memory 221 shown. The process of correcting the contrast of the second image according to the first enhancement coefficient can be implemented in a hardware manner. The function of correcting the contrast of the image can be integrated on the processor (such as the ISP) of the electronic device. The ISP can query the first enhancement coefficient stored in the internal memory, correct the contrast of the second image, and return the second image with the corrected contrast to the camera application. The camera application replaces the second image with the second image with the corrected contrast for display. The image processing method provided by the embodiments of the present application can also be implemented in a hardware manner. For example, the functions of the image processing method provided by the embodiments of the present application can be implemented on a dedicated image processor DSP or an application specific integrated circuit (ASIC), or the functions of the image processing method provided by the embodiments of the present application can be implemented on the ISP. In a possible implementation, for the process of predicting the pixel offset of the second image relative to the first image after the first image, it can also be implemented on the NPU processor.

[0287] The image processing method provided by the embodiments of the present application can also be partially implemented in a software manner and partially implemented in a hardware manner. For example, the process of predicting the pixel offset of the second image relative to the first image after the first image can be implemented in a software manner, and the processes of obtaining a first enhancement coefficient based on the first image and the pixel offset, correcting the contrast of the second image according to the first enhancement coefficient, and replacing the second image with the second image with the corrected contrast for display can all be implemented in a hardware manner.

[0288] The process of determining whether the prediction accuracy of the historical pose before the first image meets the accuracy condition can be implemented in a software or hardware manner, and the present application does not limit this.

[0289] The present application also provides an electronic device, Figure 14 showing a block diagram of the electronic device according to an embodiment of the present application. As Figure 14 shown, the electronic device provided by the embodiments of the present application may include: a camera, a processor, and a display.

[0290] The camera is used to collect a first image and send it to the processor; the first image is one of multiple frames of images continuously collected by the camera; the processor is used to predict, based on the first image, a pixel offset of a second image after the first image in the multiple frames of images relative to the first image, where the pixel offset is a spatial position change of pixels of the second image relative to the pixels of the first image; the processor is used to obtain a first enhancement coefficient based on the first image and the pixel offset; the camera is used to collect the second image and send it to the processor; the processor is used to correct the contrast of the second image based on the first enhancement coefficient; the display is used to display the corrected second image.

[0291] According to the electronic device provided by the embodiments of the present application, during the process of continuously collecting images by the camera, it predicts the pixel offset of the subsequent second image relative to the already collected first image, corrects the enhancement coefficient according to the pixel offset, and corrects the image contrast of the second image according to the corrected enhancement coefficient. By correcting the enhancement coefficient according to the spatial position change of the pixels, the electronic device according to the embodiments of the present application can minimize the spatial deviation between the enhancement coefficient and the second image, thereby reducing the inter-frame difference. Using the corrected enhancement coefficient to correct the contrast of the second image and replacing the second image with the second image with corrected contrast for display can improve the display effect.

[0292] In a possible implementation, as Figure 14 shown, the electronic device further includes a motion sensor, such as Figure 2a the gyroscope sensor 280A and the acceleration sensor 280B in the sensor module 280 part shown. The processor is used to obtain historical data of the motion sensor and predict the pixel offset of the second image relative to the first image based on the historical data and the first image, where the historical data of the motion sensor is data before collecting the second image.

[0293] In a possible implementation, the processor is further used to predict the pixel offset of the second image relative to the first image based on the information of the historical images before the second image and the first image.

[0294] In a possible implementation, the electronic device further includes a motion sensor, and the processor is further configured to obtain historical data of the motion sensor before collecting the second image, determine a motion scenario according to the historical data, and predict a pixel offset of the second image relative to the first image according to the historical motion information corresponding to the motion scenario and the first image; wherein, the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is the information of the historical image.

[0295] In a possible implementation, predicting a pixel offset of a second image after the first image in the multiple frames of images relative to the first image according to the first image includes:

[0296] Predicting a pose difference of the pose of the camera when acquiring the second image relative to the pose when acquiring the first image; and obtaining a pixel offset of the second image relative to the first image according to the pose difference.

[0297] In a possible implementation, when the historical pose prediction accuracy before the second image does not meet the accuracy condition, the processor is further configured to adjust the pose difference of the subsequent predicted image according to the pose difference, where the historical pose prediction accuracy is obtained according to the predicted pose difference and the corresponding actual pose difference before the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet.

[0298] In a possible implementation, the processor is configured to calculate a second enhancement coefficient according to the first image, correct the second enhancement coefficient according to the pixel offset, and obtain the first enhancement coefficient.

[0299] In a possible implementation, the processor is configured to correct the first image according to the pixel offset to obtain a third image, and calculate a first enhancement coefficient according to the third image.

[0300] Figure 15 A schematic diagram showing an application scenario according to an embodiment of the present application. Figure 15 compares the process of image contrast enhancement in the prior art and the process of image contrast enhancement provided in the embodiment of the present application, as Figure 15As shown, scenario (1) represents the process of image contrast enhancement in the prior art. The enhancement coefficient is calculated based on the (i - n)-th frame image, and the image contrast of the i-th frame image is directly enhanced according to the calculated enhancement coefficient. In scenario (2), the electronic device predicts the pixel offset of the second image relative to the first image, calculates the second enhancement coefficient based on the (i - n)-th frame image, performs a spatial transformation on the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient, which is stored in the internal storage module, and the ISP corrects the contrast of the second image according to the first enhancement coefficient. According to the above process, it can be seen that the image processing device provided in the embodiment of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by correcting the enhancement coefficient according to the change in the spatial position of the pixels, thereby reducing the inter-frame difference and improving the enhancement effect.

[0301] Figure 16 Fig. shows a schematic diagram of an application scenario according to an embodiment of the present application. Figure 16 compares the process of image contrast enhancement in the prior art with the process of image contrast enhancement provided in the embodiment of the present application. As Figure 16 shown, scenario (1) represents the process of image contrast enhancement in the prior art. The enhancement coefficient is calculated based on the (i - n)-th frame image, and the image contrast of the i-th frame image is directly enhanced according to the calculated enhancement coefficient. In scenario (2), the electronic device predicts the pixel offset of the i-th frame image relative to the (i - n)-th image, and performs a spatial transformation on the (i - n)-th image according to the pixel offset to obtain Figure 16 the intermediate image (the third image) shown. The first enhancement coefficient is calculated based on the intermediate image and stored in the internal storage module, and the ISP performs image contrast enhancement processing on the second image according to the first enhancement coefficient. According to the above process, it can be seen that the image processing device provided in the embodiment of the present application can minimize the spatial deviation between the enhancement coefficient and the second image by correcting the first image according to the change in the spatial position of the pixels and calculating the enhancement coefficient based on the corrected first image, thereby reducing the inter-frame difference and improving the enhancement effect.

[0302] Figure 17 Fig. shows a schematic diagram of an application scenario according to an embodiment of the present application. As Figure 17 shown, scenario (1) represents the process of image contrast enhancement in the prior art. The enhancement coefficient is calculated based on the (i - n)-th frame image, and the image contrast of the i-th frame image is directly enhanced according to the calculated enhancement coefficient. In scenario (2), the electronic device predicts the pixel offset of the second image relative to the first image, and can determine whether the historical pose prediction accuracy meets the accuracy condition. If the accuracy condition is met, the electronic device can correct the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient, and the ISP performs image contrast enhancement processing on the second image according to the first enhancement coefficient.

[0303] If the terminal determines that the historical pose prediction accuracy does not meet the accuracy condition, the electronic device can adjust the process of predicting the future image pose according to the error between the predicted pose difference and the true pose difference.

[0304] In the embodiments provided in this application, the process of predicting pixel offset can be implemented by a neural network model or other prediction models. The parameters of the neural network model or other prediction models can be adjusted according to the error between the predicted pose difference and the true pose difference feedback from the backend, so as to adjust the subsequent prediction process.

[0305] According to the above process, it can be known that the image processing device provided in the embodiments of this application can further improve the prediction accuracy by adjusting the prediction process through the feedback of the prediction result, so as to minimize the enhancement coefficient and the spatial deviation of the effective frame, thereby reducing the inter-frame difference and improving the enhancement effect.

[0306] The embodiments of this application provide an image processing device, including: a processor and a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above method when executing the instructions.

[0307] The embodiments of this application provide a non-volatile computer-readable storage medium, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0308] The embodiments of this application provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in the processor of an electronic device, the processor in the electronic device executes the above method.

[0309] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing.

[0310] The computer-readable program instructions or code described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0311] The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, including object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present application.

[0312] Aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0313] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data - processing apparatus, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause the computer, programmable data - processing apparatus, and / or other devices to operate in a specific manner. Thus, the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0314] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0315] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur in a different order than noted in the figures. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved.

[0316] It should also be noted that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, may be implemented by hardware (e.g., circuitry or an ASIC (Application Specific Integrated Circuit)) that performs the corresponding functions or acts, or may be implemented by a combination of hardware and software, such as firmware.

[0317] Although the present invention has been described in conjunction with the various embodiments, those skilled in the art will recognize other variations of the disclosed embodiments while practicing the claimed invention by viewing the figures, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the singular "a" or "an" does not exclude a plurality. A single processor or other unit may implement several functions recited in the claims. Certain measures are recited in mutually different dependent claims, but this does not indicate that these measures cannot be combined to produce a favorable effect.

[0318] The embodiments of the present application have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: acquiring a first image through a camera, where the first image is one frame of a plurality of consecutive images acquired by the camera; Based on the first image, predicting a pixel offset of a second image after the first image in the plurality of images relative to the first image, where the pixel offset is a spatial position change of pixels of the second image relative to the first image; Obtaining a first enhancement coefficient based on the first image and the pixel offset; Acquiring the second image through the camera; Correcting the contrast of the second image according to the first enhancement coefficient; Displaying the corrected second image.

2. The method according to claim 1, characterized in that, Based on the first image, predicting a pixel offset of a second image after the first image in the plurality of images relative to the first image includes: Obtaining historical data of a motion sensor, and predicting the pixel offset of the second image relative to the first image based on the historical data of the motion sensor and the first image, where the historical data of the motion sensor is data before acquiring the second image.

3. The method according to claim 1, wherein Based on the first image, predicting a pixel offset of a second image after the first image in the plurality of images relative to the first image includes: Predicting the pixel offset of the second image relative to the first image based on information of historical images before the second image and the first image.

4. The method according to claim 2, wherein Obtaining historical data of a motion sensor, and predicting the pixel offset of the second image relative to the first image based on the historical data of the motion sensor and the first image includes: Obtaining historical data of a motion sensor, and determining a motion scenario based on the historical data of the motion sensor; Predicting the pixel offset of the second image relative to the first image based on historical motion information corresponding to the motion scenario and the first image; where the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is information of historical images.

5. The method according to any one of claims 1-4, characterized in that Based on the first image, predicting a pixel offset of a second image after the first image in the plurality of images relative to the first image, includes, Predicting a pose difference of a pose of the camera when acquiring the second image relative to a pose when acquiring the first image; Obtaining the pixel offset of the second image relative to the first image based on the pose difference.

6. The method according to claim 5, wherein The method further includes: If the prediction accuracy of the historical pose before the second image does not meet the accuracy condition, then a process of adjusting the pose of a subsequent predicted image based on the pose difference, where the historical pose prediction accuracy is obtained based on a predicted pose difference and a corresponding actual pose difference before the second image, and the accuracy condition is an accuracy range that the historical pose prediction accuracy needs to meet.

7. The method according to claim 1, characterized in that, Obtaining a first enhancement coefficient based on the first image and the pixel offset includes: Calculating a second enhancement coefficient based on the first image; Correcting the second enhancement coefficient according to the pixel offset to obtain the first enhancement coefficient.

8. The method according to claim 1, characterized in that, Obtaining a first enhancement coefficient based on the first image and the pixel offset includes: Correcting the first image according to the pixel offset to obtain a third image; Calculating the first enhancement coefficient according to the third image.

9. An electronic device, characterized in that, It includes: A camera, a processor, and a display, wherein the camera is configured to collect a first image and send it to the processor; the first image is one of multiple frames of images continuously collected by the camera; The processor is configured to predict, based on the first image, a pixel offset of a second image relative to the first image among the multiple frames of images, where the pixel offset is a spatial position change of pixels of the second image relative to the first image; The processor is configured to obtain a first enhancement coefficient based on the first image and the pixel offset; The camera is configured to collect the second image and send it to the processor; The processor is configured to correct the contrast of the second image according to the first enhancement coefficient; The display is configured to display the corrected second image.

10. The electronic device according to claim 9, wherein The electronic device further includes a motion sensor, The processor is configured to obtain historical data of the motion sensor and predict, based on the historical data and the first image, a pixel offset of the second image relative to the first image, where the historical data of the motion sensor is data before collecting the second image.

11. The electronic device according to claim 9, wherein The processor is further configured to predict, based on information of historical images before the second image and the first image, a pixel offset of the second image relative to the first image.

12. The electronic device according to claim 9, characterized in that, The electronic device further includes a motion sensor, The processor is further configured to obtain historical data of the motion sensor before collecting the second image, determine a motion scenario according to the historical data, and predict, based on historical motion information corresponding to the motion scenario and the first image, a pixel offset of the second image relative to the first image; wherein the motion scenario includes a first scenario and a second scenario, the historical motion information corresponding to the first scenario is the historical data of the motion sensor, and the historical motion information corresponding to the second scenario is information of historical images.

13. The electronic device according to any one of claims 9-12, characterized in that, Predicting, based on the first image, a pixel offset of the second image relative to the first image among the multiple frames of images includes: Predicting a pose difference of the pose of the camera when obtaining the second image relative to the pose when obtaining the first image; Obtaining the pixel offset of the second image relative to the first image according to the pose difference.

14. The electronic device according to claim 13, wherein When the processor determines that the historical pose prediction accuracy before the second image does not meet the accuracy condition, the processor adjusts the pose difference of the subsequent predicted image according to the pose difference, where the historical pose prediction accuracy is obtained based on the predicted pose difference and the corresponding actual pose difference before the second image, and the accuracy condition is the accuracy range that the historical pose prediction accuracy needs to meet.

15. The electronic device according to claim 9, wherein The processor is configured to calculate a second enhancement coefficient according to the first image, correct the second enhancement coefficient according to the pixel offset, and obtain the first enhancement coefficient.

16. The electronic device according to claim 9, wherein The processor is configured to correct the first image according to the pixel offset to obtain a third image, and calculate a first enhancement coefficient according to the third image.

17. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by a processor, the method according to any one of claims 1-8 is implemented.

18. A computer program product, comprising computer-readable code, when the computer-readable code runs in an electronic device, a processor in the electronic device executes the method according to any one of claims 1-8.

19. An image processing apparatus, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, when the processor is configured to execute the instructions, the method according to any one of claims 1-8 is implemented.

Images