Image processing method, processing device and storage medium

By combining the camera module and the eye tracker, the exposure parameters are adjusted using eye tracking signals, the problem of finger operation dependence in the existing technology is solved, and automatic exposure adjustment of AR, VR, and MR devices is realized, improving user experience and image quality.

CN118921558AActive Publication Date: 2024-11-08GRAVITYXR ELECTRONICS & TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202310519382.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08
Estimated Expiration
2043-05-08

AI Technical Summary

Technical Problem

The existing automatic exposure adjustment function relies on finger operation and cannot be applied to metacosmic devices such as head-mounted displays and VR glasses that rely on eye movement signals, resulting in the inability to realize automatic exposure adjustment.

Method used

The image is acquired through the camera module, the eye tracker is used to obtain the user's eye movement signal, determine the gaze point coordinates, and adjust the exposure parameters in the image according to the coordinates to achieve automatic exposure adjustment.

Benefits of technology

It solves the problem that AR, VR, MR and other devices cannot perform automatic exposure adjustment through fingers, improves the user experience, avoids discomfort when the line of sight moves to too bright or too dark areas, and improves the comfort and clarity of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921558A_ABST
    Figure CN118921558A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method, an image processing device and a storage medium. The storage method comprises the following steps: acquiring a first image shot by a camera module through the camera module; obtaining an eye movement signal of a user to determine a staring point coordinate of the user in the first image; adjusting exposure parameters of a plurality of areas in a subsequent second image according to the staring point coordinates; and controlling the camera module to shoot the second image according to the exposure parameter. By limiting the steps, the image processing method can be used for adjusting and controlling the dynamic exposure based on the eye movement signal, so that the application requirement of augmented reality display is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, an image processing device, and a computer-readable storage medium. Background Art

[0002] Existing digital cameras and camera phones are generally equipped with an automatic exposure (AE) adjustment function. In different ambient brightness scenes, users can control the camera to automatically adjust the exposure parameters by touching the focus position with their fingers, so as to capture high-quality videos and images with appropriate brightness and darkness.

[0003] However, the existing automatic exposure adjustment function is generally based on a pre-set fixed focus position or the recognition of finger touch signals. On the one hand, it needs to rely on the user's fingers to operate, which has the defect of inconvenient operation. On the other hand, it cannot be applied to the use requirements of head-mounted displays, VR glasses and other metaverse-related devices that rely on eye movement signals for control.

[0004] In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of an image processing technology to solve the problem that extended reality display devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) cannot automatically adjust the exposure through fingers, so as to meet their usage requirements. Summary of the invention

[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing method, an image processing device, and a computer-readable storage medium, which can perform automatic exposure adjustment based on eye movement signals to solve the problem that AR, VR, MR and other extended reality display devices cannot perform automatic exposure adjustment through fingers, thereby adapting to their usage requirements.

[0007] Specifically, the above-mentioned image processing method provided according to the first aspect of the present invention includes the following steps: obtaining a first image taken by a camera module; obtaining the user's eye movement signal to determine the coordinates of his gaze point in the first image; adjusting the exposure parameters of multiple areas in a subsequent second image according to the gaze point coordinates; and controlling the camera module to take the second image according to the exposure parameters.

[0008] Furthermore, in some embodiments of the present invention, the step of acquiring the first image captured by the camera module includes: acquiring the original images captured by the multiple camera modules respectively; and preprocessing each of the original images to obtain the first image.

[0009] Further, in some embodiments of the present invention, the step of preprocessing each of the original images to obtain the first image includes: performing image analysis on each of the original images to determine the common picture of each of the original images, and determining the first image based on the image data of the common picture; or performing image analysis on each of the original images to determine the picture content of each of the original images, and selecting one of the original images as the first image based on the picture content; or performing image analysis on each of the original images to determine the difference picture of each of the original images, and splicing each of the original images based on the image data of the difference picture to determine the first image.

[0010] Furthermore, in some embodiments of the present invention, the camera module is installed in a head-mounted device. The head-mounted device is configured with a display screen and an eye tracker. The step of obtaining the user's eye movement signal to determine the coordinates of the gaze point in the first image includes: de-distorting the first image according to the difference between the first resolution of the first image and the second resolution of the display screen, and displaying the obtained third image on the display screen; obtaining the user's eye movement signal via the eye tracker, and determining the first gaze point coordinates of the user in the third image according to the eye movement signal; and performing an inverse change of the de-distortion process on the first gaze point coordinates to determine the second gaze point coordinates of the user in the first image.

[0011] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in a subsequent second image according to the gaze point coordinates includes: recording the exposure parameters of multiple areas in the first image in a light meter, wherein the light meter includes multiple cells, each cell corresponding to one of the areas in the first image; and performing a weighted operation on the exposure parameters in at least one cell of the light meter according to the gaze point coordinates to update the light meter.

[0012] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: determining the maximum difference in pixel grayscale values ​​of the gaze point coordinates of the first image continuously shot by the user in multiple frames; in response to the maximum difference being less than a preset first difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates; and in response to the maximum difference being greater than or equal to the first difference threshold, maintaining the exposure parameters of each of the areas in the first image.

[0013] Further, in some embodiments of the present invention, in response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the maximum difference being less than the first difference threshold, determining a convergence speed for adjusting the exposure parameters according to the maximum difference; and adjusting the exposure parameters of each of the areas in each of the subsequent second images frame by frame according to the convergence speed.

[0014] Further, in some embodiments of the present invention, the step of adjusting the exposure parameters of each of the areas in each of the subsequent second images frame by frame according to the convergence speed includes: corresponding to a larger first maximum difference, adjusting the exposure parameters of each of the areas in each of the second images frame by frame with a smaller first convergence speed; and corresponding to a smaller second maximum difference, adjusting the exposure parameters of each of the areas in each of the second images frame by frame with a larger second convergence speed.

[0015] Furthermore, in some embodiments of the present invention, the step of determining the maximum difference in pixel grayscale values ​​of the gaze point coordinates of the user in multiple frames of continuously captured first images includes: caching the pixel grayscale values ​​of the gaze point coordinates of the multiple frames of continuously captured first images; and in response to capturing a new frame of first image, replacing the cached pixel grayscale values ​​of the gaze point coordinates of the first frame of the first image with the pixel grayscale values ​​of the gaze point coordinates of the new first image, and re-determining the maximum difference.

[0016] Further, in some embodiments of the present invention, in response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the re-determined maximum difference being less than the first difference threshold, determining the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last two frames of the first image; in response to the difference being less than a preset second difference threshold, maintaining the exposure parameters of each of the areas in the first image; and in response to the difference being greater than or equal to the second difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates.

[0017] Further, in some embodiments of the present invention, in response to the difference being greater than or equal to the second difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the difference being greater than or equal to the second difference threshold and less than a preset third difference threshold, performing a weighted operation on the exposure parameters of the last frame and the last two frames of the first image to determine the exposure parameters of each of the areas in the second image; and in response to the difference being greater than or equal to the third difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the exposure parameters of the gaze point area in the last frame of the first image.

[0018] Further, in some embodiments of the present invention, in response to the re-determined maximum difference being less than the first difference threshold, the step of determining the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last two frames of the first image includes: in response to the re-determined maximum difference being less than the first difference threshold, determining the gaze point area of ​​the user in the last frame and the last two frames of the first image respectively according to the gaze point coordinates; and in response to a judgment result that the gaze point area of ​​the user in the last frame and the last two frames of the first image has changed, calculating the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last frame and the last two frames of the first image.

[0019] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: determining the pixel grayscale value of the user's gaze point coordinates in the first image; in response to the pixel grayscale value being greater than a preset grayscale value lower limit and less than a preset grayscale value upper limit, maintaining the exposure parameters of each of the areas in the first image; and in response to the pixel grayscale value being less than the grayscale value lower limit or greater than the grayscale value upper limit, adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates.

[0020] Furthermore, in some embodiments of the present invention, the image processing method also includes the following steps: in response to the pixel grayscale value being less than the grayscale value lower limit, reducing the grayscale value upper limit and the grayscale value lower limit according to a preset relaxation step, wherein the relaxation step is less than the difference between the grayscale value upper limit and the grayscale value lower limit; and in response to the pixel grayscale value being greater than the grayscale value upper limit, increasing the grayscale value upper limit and the grayscale value lower limit according to the relaxation step.

[0021] Furthermore, in some embodiments of the present invention, the image processing method further includes the following steps: displaying the first image via a display screen; and displaying a corresponding second image according to a refresh rate of the display screen to refresh the first image.

[0022] In addition, the image processing device provided according to the second aspect of the present invention includes a camera module, a memory and a processor. The memory stores computer instructions. The processor is communicatively connected to the camera module and the memory, and is configured to execute the computer instructions stored in the memory to implement the image processing method as described in any one of the first aspects of the present invention.

[0023] Furthermore, in some embodiments of the present invention, the image processing device further comprises a display screen and an eye tracker. The display screen is used to display the first image and / or the second image captured by the camera module. The eye tracker is used to capture the user's eye image to obtain the user's eye movement signal.

[0024] In addition, the computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method as described in any one of the first aspects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0026] Figure 1 A schematic diagram of the architecture of an image processing device provided according to some embodiments of the present invention is shown.

[0027] Figure 2 A schematic flow chart of an image processing method provided according to some embodiments of the present invention is shown.

[0028] Figure 3 A schematic diagram of gaze point coordinate conversion provided according to some embodiments of the present invention is shown.

[0029] Figure 4 A schematic diagram of a process for adjusting exposure parameters provided according to some embodiments of the present invention is shown.

[0030] Figure 5 A curve diagram showing the corresponding relationship between exposure parameter differences and weighting coefficients provided according to some embodiments of the present invention is shown.

[0031] Figure 6 A schematic diagram of a light meter provided according to some embodiments of the present invention is shown.

[0032] Figure 7 A schematic diagram of a process for adjusting exposure parameters according to some other embodiments of the present invention is shown. DETAILED DESCRIPTION

[0033] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0034] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0036] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0037] As mentioned above, the existing automatic exposure (AE) adjustment function is generally implemented based on a pre-set fixed focus position or the recognition of finger touch signals. On the one hand, it needs to rely on the user's fingers to operate, which has the defect of inconvenient operation. On the other hand, it cannot be applied to the use requirements of head-mounted displays, VR glasses and other metaverse-related devices that rely on eye movement signals for control.

[0038] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing method, an image processing device, and a computer-readable storage medium, which can automatically adjust the exposure based on eye movement signals, thereby solving the problem that extended reality display devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) cannot automatically adjust the exposure through fingers, so as to meet their usage requirements.

[0039] In some non-limiting embodiments, the image processing method provided by the first aspect of the present invention can be implemented via the image processing device provided by the second aspect of the present invention. Specifically, the image processing device is configured with a memory and a processor. The memory includes but is not limited to the above-mentioned computer-readable storage medium provided by the third aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement any one of the image processing methods described in the first aspect of the present invention.

[0040] Please refer to Figure 1 , Figure 1 A schematic diagram of the architecture of an image processing device provided according to some embodiments of the present invention is shown.

[0041] like Figure 1 As shown, the image processing device provided by the second aspect of the present invention may be configured with a camera module 10, a memory (not shown), a processor 20, a display screen 30 and an eye tracker 40. Here, the image processing device may be a head-mounted device such as extended reality (XR) display glasses / mask / helmet, or a camera device such as a digital camera or a smart phone.

[0042] For example, in a head-mounted device such as an MR helmet, the camera module 10 may include a left-eye real-view camera for collecting images of the user's left eye, and a right-eye real-view camera for collecting images of the user's right eye. The processor 20 may include a graphics processing unit (GPU) for acquiring original images, generating virtual images and fused images. The display screen 30 may be integrated into the MR helmet to display real-view images, virtual images and / or fused images of the two to the user. The eye tracker 40 may be integrated into the MR helmet to collect eye movement signals of the user's left and right eyes respectively, as a processing basis for subsequent MR display and eye movement automatic exposure adjustment.

[0043] For another example, in a camera device such as a smart phone, the camera module 10 may include a rear camera (group) disposed on the back of the phone, for capturing the real scene image behind the phone to be captured. The display screen 30 may be a main display interface disposed on the front of the phone, for displaying the image currently captured by the camera module 10 to the user. The eye tracker 40 may be a front camera also disposed on the front of the phone, for capturing the eye movement signal of the user observing the display screen 30, as a processing basis for subsequent eye movement automatic exposure adjustment.

[0044] The working principle of the above-mentioned image processing device will be described below in conjunction with some embodiments of the image processing method. Those skilled in the art will understand that the embodiments of these processing methods are only some non-limiting implementation methods provided by the present invention, and are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working methods of the processing device. Similarly, the processing device is also only some non-limiting implementation methods provided by the present invention, and does not constitute a limitation on the execution subject or execution order of each step in these processing methods.

[0045] Please refer to Figure 1 and Figure 2 , Figure 2 A schematic flow chart of an image processing method provided according to some embodiments of the present invention is shown.

[0046] like Figure 1 and Figure 2 As shown, in the process of automatic exposure adjustment based on the eye movement signal, the processor 20 can first obtain the first image taken by the camera module 10.

[0047] Specifically, continuing to take the above-mentioned MR helmet as an example, in the process of acquiring the first image taken by the camera module 10, the processor 20 can first acquire the left-eye original (raw) image and the right-eye original image taken by the left and right real-scene cameras of the camera module 10, respectively, and pre-process them according to a pre-configured processing strategy to integrate the original images captured by each real-scene camera into a first image.

[0048] For example, based on a preset first strategy, the processor 20 may perform image analysis on the original images captured by the left and right real-view cameras to determine a common picture of the original images, and then determine the first image based on the image data of the common picture.

[0049] For another example, based on the preset second strategy, the processor 20 can perform image analysis on the original images captured by the left and right real-view cameras to respectively determine the picture content of each original image, and select an original image that better meets the eye movement automatic exposure adjustment requirements as the first image based on the picture content.

[0050] For another example, based on the preset third strategy, the processor 20 can perform image analysis on each original image captured by the left and right real-view cameras to determine the difference pictures of each original image, and perform operations such as upsampling, downsampling, and rotation on each original image to unify the resolution and image angle according to the image data of each difference picture, so as to splice each original image into a complete first image. Furthermore, the processor 20 can also perform downsampling processing on the spliced ​​first image to make it meet the resolution requirement of the eye movement automatic exposure adjustment for the first image.

[0051] In this way, by unifying the original images captured by multiple cameras into a first image, the present invention can effectively avoid the problem of inconsistent brightness of images taken by binocular real-scene cameras, thereby improving the binocular consistency of image processing and improving the comfort of user observation.

[0052] Please continue to refer to Figure 1 and Figure 2 After completing the preprocessing operation on the original image and obtaining the first image captured by the original image, the processor 20 transmits the first image to the image signal processing (ISP) pipeline of the image processing device (e.g., MR helmet), performs ISP processing on the first image, and then displays it on the display screen 30. After that, the processor 20 can obtain the eye movement signal of the user observing the display screen 30 through the eye tracker 40 to determine the coordinates of the gaze point in the first image.

[0053] For details, please refer to Figure 3 , Figure 3 A schematic diagram of gaze point coordinate conversion provided according to some embodiments of the present invention is shown.

[0054] like Figure 3 As shown, in the process of determining the coordinates of the user's gaze point in the first image, the present invention can first use the Dewarp algorithm to dedistort the first image according to the difference between the first resolution of the first image and the second resolution of the display screen 30, and display the obtained third image on the display screen 30. Afterwards, the processor 20 can obtain the user's eye movement signal through the eye tracker 40, and determine the user's first gaze point coordinates (x, y) in the third image according to the eye movement signal. Afterwards, the processor 20 can obtain the relevant parameters of the above dedistortion processing, and perform the inverse change of the dedistortion processing on the first gaze point coordinates (x, y) according to the parameters to determine the user's second gaze point coordinates (x', y') in the first image.

[0055] Please continue to refer to Figure 1 and Figure 2After determining the user's gaze point coordinates (x', y') in the first image, the processor 20 can input the gaze point coordinates (x', y') and the pixel grayscale values ​​of multiple areas in the first image into an automatic exposure (AE) adjustment algorithm based on eye movement signals, and adjust the exposure parameters of multiple areas in the subsequently captured second image according to the gaze point coordinates (x', y') through the AE adjustment algorithm to avoid discomfort caused by the user observing overly bright or dark areas, thereby bringing a better viewing experience to the user.

[0056] Please refer to Figure 4 , Figure 4 A schematic diagram of a process for adjusting exposure parameters provided according to some embodiments of the present invention is shown.

[0057] like Figure 4 As shown, in the process of adjusting the exposure parameters, the AE adjustment algorithm can first determine whether the pixel grayscale value of the user's gaze point coordinates is stable, and adjust the exposure parameters only after determining that the pixel grayscale value of the user's gaze point coordinates is stable.

[0058] In some embodiments, the AE adjustment algorithm may first determine the maximum difference X of the pixel grayscale values ​​of the user's gaze point coordinates in the first image continuously captured in multiple frames. max -X min and compare it with the preset first difference threshold X th1 The comparison is performed to determine whether the exposure parameters of the second image captured subsequently need to be adjusted.

[0059] Specifically, in response to the maximum difference being less than a preset first difference threshold (ie, X max -X min <X th1 ), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is stable, thereby adjusting the exposure parameters of each area in the second image according to the gaze point coordinates. max -X min ≥X th1 ), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is unstable, thereby maintaining the exposure parameters of each area in the first image.

[0060] In this way, the present invention can better adapt to the characteristics of rapid and continuous movement of the gaze point, by limiting the first difference threshold X th1 , to avoid unnecessary AE adjustment of multiple frames of images during the rapid and continuous movement of the gaze point, and instead start AE adjustment after the gaze point reaches the vicinity of the target position, thereby reducing the data processing load of the image processing device.

[0061] Furthermore, in order to prevent the human eye from perceiving the difference in brightness jump during the process of adjusting the exposure parameters, the AE adjustment algorithm may also preferably be configured based on the maximum difference X max -X min To determine the convergence speed of adjusting the exposure parameters, and adjust the exposure parameters of each area in the multiple second images taken subsequently frame by frame according to the convergence speed. Specifically, corresponding to a larger first maximum difference, the AE adjustment algorithm can use a smaller first convergence speed to adjust the exposure parameters of each area in each subsequent second image frame by frame, so as to avoid the human eye from perceiving the difference in brightness jumps during the process of adjusting the exposure parameters. Conversely, corresponding to a smaller second maximum difference, the AE adjustment algorithm can use a larger second convergence speed to adjust the exposure parameters of each area in each subsequent second image frame by frame, thereby improving the response rate of the AE adjustment function while ensuring observation comfort.

[0062] Furthermore, in determining the maximum difference X max -X min In the process, the AE adjustment algorithm can use an array iteration method to cache and iterate the pixel grayscale value X of the gaze point coordinates of the first image of the continuously shot n frames. i ~X i+n-1 , and update the maximum difference value X of the first image of each frame accordingly max -X min Specifically, in response to acquiring a new first image, the AE adjustment algorithm may first delete the pixel grayscale value X of the first frame of the first image cached in the array. i , and the pixel grayscale value X of the first image of the remaining frames i+1 ~X i+n-1 All of them are moved forward one position, and then the pixel gray value X of the newly acquired first image is i+n Cache to the end of the array. In this way, the AE adjustment algorithm can use the pixel grayscale value X of the gaze point coordinates of the new first image i+n , replace the pixel gray value X of the gaze point coordinates of the first frame of the first image in the cache i , and redetermine the maximum difference in the array.

[0063] Then, in response to the re-determined maximum difference being less than the first difference threshold X th1 , the AE adjustment algorithm can determine the maximum difference from the first difference threshold X th1The above falls back to the lower state, that is, the pixel grayscale value of the user's gaze point coordinates recovers from an unstable state to a stable state, so that the user's gaze point area in the last two frames of the first image is preferably determined according to the gaze point coordinates. In response to the gaze point area moving from the first area of ​​the last two frames of the image to the second area of ​​the last frame of the image, the AE adjustment algorithm can determine that there is a large gaze point displacement in the last two frames of the first image, and further calculate the difference in the pixel grayscale value of the user's gaze point coordinates in the last frame and the last two frames of the first image to determine whether it is necessary to adjust the exposure parameters of the second image captured subsequently.

[0064] On the contrary, in response to the user's gaze point area in the last two frames of the first image remaining unchanged, the AE adjustment algorithm can determine that there is no sufficiently large gaze point displacement in the last two frames of the first image, thereby continuing to maintain the exposure parameters of each area in the first image.

[0065] In this way, the present invention can better adapt to the characteristics of frequent gaze point movement and poor stability, and avoid excessive AE adjustment of multiple frames of images with too small gaze point coordinate changes by limiting the change threshold of the gaze point coordinate position, thereby reducing the data processing load of the image processing device.

[0066] Furthermore, the AE adjustment algorithm can compare the difference X1-X2 of the pixel grayscale value of the user's gaze point coordinates in the last two frames of the first image with the preset second difference threshold X th2 The second difference threshold X is compared to determine whether the exposure parameters of the second image taken subsequently need to be adjusted. th2 It can be compared with the first difference threshold X th1 has the same value, and can also be the same as the first difference threshold X th1 Has different values.

[0067] Specifically, in response to the difference being less than a preset second difference threshold (ie, X1-X2 <X th2 ), the AE adjustment algorithm can determine that the difference in the brightness of the gaze point in the last two frames of the first image is too small, thereby maintaining the exposure parameters of each area in the first image. On the contrary, in response to the above difference being greater than or equal to the second difference threshold (i.e., X1-X2≥X th2 ), the AE adjustment algorithm can determine that the brightness of the gaze point in the last two frames of the first image has a large difference, and thus adjust the exposure parameters of each area in the second image taken subsequently according to the above gaze point coordinates.

[0068] In this way, the present invention can better adapt to the characteristics of frequent and continuous movement of the gaze point, by limiting the second difference threshold X th2 This is to avoid excessive AE adjustment for multiple frames of images with very small differences in gaze point brightness changes, thereby reducing the data processing load of the image processing device.

[0069] Furthermore, the AE adjustment algorithm may also be preferably configured with a third difference threshold value X th3 The AE adjustment algorithm can preferably be based on the third difference threshold X th3 , performing a weighted operation on the exposure parameters of at least one area in the last frame and the last two frames of the first image to adjust the exposure parameters of the corresponding areas in the second image captured subsequently.

[0070] Please refer to Figure 4 and Figure 5 , Figure 5 A curve diagram showing the corresponding relationship between exposure parameter differences and weighting coefficients provided according to some embodiments of the present invention is shown.

[0071] like Figure 4 and Figure 5 As shown, in the process of performing weighted operation on the exposure parameters, the AE adjustment algorithm can firstly compare the above X1-X2 difference with the second difference threshold X th2 and the third difference threshold X th3 In response to the difference being greater than or equal to the second difference threshold and less than the third difference threshold (ie, X th2 ≤X1-X2 <X th3 ), the AE adjustment algorithm can be queried based on the actual difference Figure 5 The corresponding relationship curve diagram shown is used to determine the corresponding weighting coefficient w1, and then the exposure parameters of the last frame and the last two frames of the first image are weighted according to the weighting coefficient w1 to determine the exposure parameters of each area in the second image.

[0072] Specifically, the calculation formula for performing weighted operation on the exposure parameters of the last frame and the last two frames of the first image according to the weighted coefficient w1 can be as follows:

[0073] T=T n-1 (1-w1)+T n w1

[0074] Among them, T n-1 represents the exposure parameter of the first image of the last two frames, T n represents the exposure parameter of the first image of the last frame, and w1 represents the weighting coefficient.

[0075] On the contrary, in response to the difference being greater than or equal to the third difference threshold (X1-X2≥X th3 ), the AE adjustment algorithm can be based on Figure 5 The corresponding relationship curve diagram shown determines the weighting coefficient w1=1, so that the exposure parameters of each area in the second image are adjusted completely according to the exposure parameters of the gaze point area in the last frame of the first image.

[0076] Furthermore, after determining the weighting coefficient w1 of each first image involved in the weighted operation, the AE adjustment algorithm can also determine the weighting coefficient w2 of multiple areas in the first image according to the above-mentioned gaze point coordinates (x', y'), and adjust the exposure parameters of multiple areas in the subsequent second image according to the weighting coefficient w2.

[0077] Please refer to Figure 6 , Figure 6 A schematic diagram of a light meter provided according to some embodiments of the present invention is shown.

[0078] like Figure 6 As shown, in the process of adjusting the exposure parameters of multiple areas in the second image according to the gaze point coordinates (x', y'), the AE adjustment algorithm can first divide the entire picture of the first image into several small areas, and count the grayscale value of each area. After that, the AE adjustment algorithm can record the grayscale values ​​of multiple areas in the first image in multiple cells in the light meter, assign a corresponding weight w2∈[0,1] to each cell according to the gaze point coordinates (x', y'), and then perform a weighted operation on the pixel grayscale value in at least one cell of the light meter according to the weight w2 to determine an updated light meter.

[0079] Specifically, each cell in the light meter corresponds to an area in the first image. The AE adjustment algorithm can calculate the distance from the center (a, b), (x', y'), (c, d) of each area to the gaze point coordinate (x', y'), set the weight w2 = 1 for the closest gaze point area, and set the weight w2 = 0 for the farthest non-gaze point area.

[0080] Furthermore, in some embodiments, for a large light meter including areas of various distances, the AE adjustment algorithm may also configure weights w2∈(0,1) for multiple areas of equal distance so that they evenly share the total weight (ie, 1).

[0081] After that, if Figure 1 and Figure 2 As shown, the AE adjustment algorithm can calculate the grayscale value of the entire picture according to the weight w2 of each area, calculate the corresponding illumination value according to the grayscale value of the entire picture, and send the corresponding exposure parameters to the camera module 10 to control the camera module 10 to continue to shoot the subsequent second image.

[0082] Those skilled in the art will understand that Figure 4 The examples shown in the figure for adjusting the exposure parameters of the subsequent second images according to the difference in pixel grayscale values ​​between multiple frames of first images are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, but are not intended to limit the scope of protection of the present invention.

[0083] Optionally, in some other embodiments, the AE adjustment algorithm can also achieve the effect of dynamically adjusting the exposure parameters of multiple regions in the subsequent captured second image by setting a grayscale value threshold.

[0084] For details, please refer to Figure 7 , Figure 7 which shows a schematic flowchart of adjusting exposure parameters according to some other embodiments of the present invention.

[0085] As Figure 7 shown, in response to determining the gaze point coordinates of the user in the first image, the AE adjustment algorithm can first determine the pixel grayscale value of the gaze point coordinates of the user in the first image and compare it with a preset lower grayscale value threshold Th0 and an upper grayscale value threshold Th1. In response to the judgment result that the pixel grayscale value of the gaze point coordinates is between the lower grayscale value threshold Th0 and less than the upper grayscale value threshold Th1 (i.e., Th0 ≤ X ≤ Th1), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is stable, thereby maintaining the exposure parameters of each region in the first image. Conversely, in response to the judgment result that the pixel grayscale value of the gaze point coordinates exceeds the lower grayscale value threshold Th0 or the upper grayscale value threshold Th1 (i.e., X < Th0 or X > Th1), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is unstable, and thus adjust the exposure parameters of each region in the second image as described above according to the gaze point coordinates.

[0086] Furthermore, in order to avoid too frequent jumps in exposure parameters, the AE adjustment algorithm can also dynamically adjust the above-mentioned lower grayscale value threshold Th0 and upper grayscale value ΔTh according to the jump direction of the exposure parameters to reduce the data processing load of the image processing device.

[0087] Specifically, in response to the judgment result that the pixel grayscale value of the gaze point coordinates is less than the lower grayscale value (i.e., X < Th0), the AE adjustment algorithm can, while adjusting the exposure parameters of each region in the second image, reduce the original lower grayscale value threshold Th0 and upper grayscale value Th1 according to a preset relaxation step size ΔTh. Conversely, in response to the judgment result that the pixel grayscale value of the gaze point coordinates is greater than the upper grayscale value (i.e., X > Th1), the AE adjustment algorithm can also increase the original lower grayscale value threshold Th0 and upper grayscale value Th1 according to this relaxation step size ΔTh. Here, the relaxation step size can be less than the difference between the upper grayscale value and the lower grayscale value (i.e., ΔTh < Th1 - Th0), so as to avoid the adjusted lower grayscale value threshold Th0' being greater than the original upper grayscale value Th1 and avoid the adjusted upper grayscale value threshold Th1' being less than the original lower grayscale value Th0, thereby avoiding the exposure parameters from jumping again in a short time through the relaxation change method.

[0088] In this way, the AE adjustment algorithm can achieve the effect of dynamically adjusting the exposure parameters of multiple areas in the second image captured subsequently by setting and adjusting the gray value threshold.

[0089] Furthermore, if Figure 1 and Figure 2 As shown, for head-mounted devices such as extended reality display glasses / masks / helmets and photographic devices such as digital cameras and smart phones with a display screen 30, the processor 20 can also continue to transmit the captured second image to the display screen 30 after displaying the first image via the display screen 30, thereby controlling the display screen 30 to display the corresponding second image according to its refresh frequency to refresh the previous first image.

[0090] In summary, by determining the coordinates of the user's gaze point in the first image according to the eye movement signal, and then adjusting the exposure parameters of multiple areas in the subsequently captured second image according to the gaze point coordinates, the above-mentioned image processing method, image processing device and computer-readable storage medium provided by the present invention can not only dynamically adjust the exposure parameters of the camera by following the movement of the user's line of sight to solve the problem that AR, VR, MR and other extended reality display devices cannot automatically adjust the exposure through fingers and adapt to their usage needs, but also avoid the glare phenomenon caused by the user moving the line of sight to an overly bright area, and the black screen phenomenon caused by the user moving the line of sight to an overly dark area, thereby improving the user's comfort and clarity in observing the image.

[0091] Those skilled in the art will understand that Figure 1 The embodiments of the image processing device shown, which also include a camera module 10, a memory (not shown), a processor 20, a display screen 30 and an eye tracker 40, are only some non-limiting implementation methods provided by the present invention, and are intended to clearly demonstrate the main concepts of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit the scope of protection of the present invention.

[0092] Optionally, in other embodiments, the image processing device does not need to be configured with the above-mentioned display screen 30 and / or eye tracker 40, but can also achieve the above-mentioned eye movement automatic exposure adjustment function by connecting to an external display screen and / or an external eye tracker.

[0093] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0094] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0095] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0096] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0097] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0098] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: The following steps are involved: Acquiring a first image captured by the camera module; Acquire an eye movement signal of the user to determine the coordinates of the gaze point of the user in the first image; adjusting exposure parameters of a plurality of regions in a subsequent second image according to the gaze point coordinates; as well as According to the exposure parameters, the camera module is controlled to capture the second image.

2. The image processing method according to claim 1, characterized in that: The step of acquiring the first image captured by the camera module comprises: Acquiring original images captured by the plurality of camera modules respectively; and Preprocess each of the original images to obtain the first image.

3. The image processing method according to claim 2, characterized in that: The step of preprocessing each of the original images to obtain the first image comprises: Performing image analysis on the original images to determine a common frame of the original images, and determining the first image according to image data of the common frame; or Performing image analysis on each of the original images to determine the picture content of each of the original images, and selecting one of the original images as the first image according to the picture content; or Image analysis is performed on each of the original images to determine a difference picture of each of the original images, and each of the original images is spliced ​​according to image data of the difference picture to determine the first image.

4. The image processing method according to claim 1, wherein: The camera module is installed in a head mounted device, and the head mounted device is provided with a display screen and an eye tracker. The step of obtaining the user's eye movement signal to determine the coordinates of the gaze point in the first image includes: performing dedistortion processing on the first image according to a difference between a first resolution of the first image and a second resolution of the display screen, and displaying the acquired third image on the display screen; Acquiring an eye movement signal of the user via the eye tracker, and determining the coordinates of a first gaze point of the user in the third image according to the eye movement signal; and An inverse change of the de-distortion process is performed on the first gaze point coordinates to determine the second gaze point coordinates of the user in the first image.

5. The image processing method according to claim 1, wherein: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Recording the grayscale values ​​of the plurality of regions in the first image in a light meter, wherein the light meter includes a plurality of cells, each of the cells corresponding to one of the regions in the first image; and According to the gaze point coordinates, a weighted operation is performed on the grayscale value in at least one cell of the light meter to update the light meter.

6. The image processing method according to claim 1, wherein: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Determine a maximum difference in pixel grayscale values ​​of the gaze point coordinates of the user in the first image continuously shot in multiple frames; In response to the maximum difference being less than a preset first difference threshold, adjusting the exposure parameter of each of the regions in the second image according to the gaze point coordinates; and In response to the maximum difference being greater than or equal to the first difference threshold, maintaining the exposure parameter of each of the regions in the first image.

7. The image processing method according to claim 6, characterized in that: In response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the maximum difference being less than the first difference threshold, determining a convergence speed for adjusting the exposure parameter according to the maximum difference; and According to the convergence speed, the exposure parameter of each of the regions in each of the subsequent second images is adjusted frame by frame.

8. The image processing method according to claim 7, characterized in that: The step of adjusting the exposure parameters of each of the regions in each of the subsequent second images frame by frame according to the convergence speed includes: corresponding to the larger first maximum difference, adjusting the exposure parameter of each of the regions in each of the second images frame by frame at a smaller first convergence speed; and Corresponding to the smaller second maximum difference, the exposure parameter of each of the regions in each of the second images is adjusted frame by frame at a larger second convergence speed.

9. The image processing method according to claim 6, characterized in that: The step of determining the maximum difference in pixel grayscale values ​​of the gaze point coordinates of the first image continuously shot by the user in multiple frames comprises: Buffering the pixel grayscale values ​​of the gaze point coordinates of the plurality of frames of first images captured continuously; and In response to acquiring a new first image frame, the pixel grayscale values ​​of the gaze point coordinates of the cached first frame first image are replaced with the pixel grayscale values ​​of the gaze point coordinates of the new first image, and the maximum difference is re-determined.

10. The image processing method according to claim 9, characterized in that: In response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the re-determined maximum difference being less than the first difference threshold, determining a difference in pixel grayscale values ​​of the gaze point coordinates of the user in the last two frames of the first image; In response to the difference being less than a preset second difference threshold, maintaining the exposure parameter of each of the regions in the first image; and In response to the difference being greater than or equal to the second difference threshold, adjusting the exposure parameter of each of the regions in the second image according to the gaze point coordinates.

11. The image processing method according to claim 10, characterized in that: In response to the difference being greater than or equal to the second difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the difference being greater than or equal to the second difference threshold and less than a preset third difference threshold, performing a weighted operation on the exposure parameters of the last frame and the last two frames of the first image to determine the exposure parameters of each of the areas in the second image; and In response to the difference being greater than or equal to the third difference threshold, the exposure parameters of each of the regions in the second image are adjusted according to the exposure parameters of the gaze point region in the last frame of the first image.

12. The image processing method according to claim 10, characterized in that: In response to the re-determined maximum difference being less than the first difference threshold, the step of determining the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last two frames of the first image comprises: In response to the re-determined maximum difference being less than the first difference threshold, determining the gaze point regions of the user in the last frame and the last two frames of the first image respectively according to the gaze point coordinates; and In response to a determination result that the gaze point area of ​​the user in the last frame and the last two frames of the first image changes, a difference in pixel grayscale values ​​of the gaze point coordinates of the user in the last frame and the last two frames of the first image is calculated.

13. The image processing method according to claim 1, characterized in that: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Determining a pixel grayscale value of a gaze point coordinate of the user in the first image; In response to the pixel grayscale value being greater than a preset grayscale value lower limit and less than a preset grayscale value upper limit, maintaining the exposure parameter of each of the areas in the first image; and In response to the pixel grayscale value being less than the grayscale value lower limit or greater than the grayscale value upper limit, the exposure parameter of each of the regions in the second image is adjusted according to the gaze point coordinates.

14. The image processing method according to claim 13, characterized in that: The following steps are also included: In response to the pixel grayscale value being less than the grayscale value lower limit, reducing the grayscale value upper limit and the grayscale value lower limit according to a preset relaxation step length, wherein the relaxation step length is less than the difference between the grayscale value upper limit and the grayscale value lower limit; as well as In response to the pixel grayscale value being greater than the grayscale value upper limit, the grayscale value upper limit and the grayscale value lower limit are increased according to the relaxation step length.

15. The image processing method according to claim 1, characterized in that: The following steps are also included: displaying the first image via a display screen; and The corresponding second image is displayed according to the refresh frequency of the display screen to refresh the first image.

16. An image processing device, characterized in that: include: Camera module; a memory having computer instructions stored thereon; as well as A processor is communicatively connected to the camera module and the memory, and is configured to execute computer instructions stored in the memory to implement the image processing method according to any one of claims 1 to 15.

17. The image processing device according to claim 16, wherein: Also includes: A display screen, used to display the first image and / or the second image captured by the camera module; as well as An eye tracker is used to capture eye images of a user to obtain eye movement signals of the user.

18. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.

Citation Information

Patent Citations

  • Image stitching for three-dimensional video

    CN107079141A

  • Display drive circuit and drive method thereof and display device

    CN107742512A

  • Eye gaze tracking correction using pursuit vector

    CN111226188A

  • Display method, device and system, storage medium and displayer

    CN112819967A

  • Display method and device, head-mounted display equipment and storage medium

    CN115883816A