Image processing method and processing apparatus, and storage medium
By acquiring and analyzing user's eye movement signals, adjusting the exposure parameters in the images captured by the camera module, the problem of automatic exposure adjustment relies on finger operation in the prior art is solved, and automatic exposure adjustment of AR, VR, and MR devices is realized , improves the user experience.
Patent Information
- Application Number
- PCT/CN2024/089776
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-14
AI Technical Summary
The existing automatic exposure adjustment function relies on finger operation and cannot be applied to metacosmic devices such as head-mounted displays and VR glasses that rely on eye movement signals, resulting in the inability to realize automatic exposure adjustment.
By acquiring the images captured by the camera module, determining the user's eye movement signal and gaze point coordinates, adjusting the exposure parameters in the subsequent images, and realizing automatic exposure adjustment based on the eye movement signal.
It solves the problem that extended reality devices such as AR, VR, and MR cannot perform automatic exposure adjustment through fingers, improves user observation comfort and image clarity, and adapts to the use needs of metacosmic equipment.
Smart Images

Figure CN2024089776_14112024_PF_FP_ABST
Abstract
Description
Image processing method, processing device and storage medium
[0001] This invention claims priority to a patent application with a filing date of May 8, 2023, Chinese application number 202310519382.8, and titled “An image processing method, processing device and storage medium”. Technical Field
[0002] The present invention relates to the field of image processing technology, and in particular to an image processing method, an image processing device, and a computer-readable storage medium. Background Art
[0003] Current digital cameras and camera phones are commonly equipped with an automatic exposure (AE) function. Users can control the camera to automatically adjust exposure parameters in different ambient brightness scenarios by simply touching the focus point with their finger, thereby capturing high-quality videos and images with appropriate brightness and darkness.
[0004] However, the existing automatic exposure adjustment function is generally based on a pre-set fixed focus position or the recognition of finger touch signals. On the one hand, it needs to rely on the user's fingers to operate, which has the defect of being inconvenient to operate. On the other hand, it cannot be applied to the use requirements of metaverse-related devices such as head-mounted displays and VR glasses that rely on eye movement signals for control.
[0005] In order to overcome the above-mentioned defects of the existing technology, this field urgently needs an image processing technology to solve the problem that extended reality display devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) cannot automatically adjust the exposure through fingers, so as to adapt to their usage requirements.
[0006] Summary of the Invention
[0007] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.
[0008] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing method, an image processing device, and a computer-readable storage medium, which can automatically adjust exposure based on eye movement signals to solve the problem that AR, VR, MR and other extended reality display devices cannot automatically adjust exposure through fingers, thereby adapting to their usage requirements.
[0009] Specifically, the above-mentioned image processing method provided according to the first aspect of the present invention includes the following steps: obtaining a first image captured by a camera module; obtaining the user's eye movement signal to determine the coordinates of his gaze point in the first image; adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates; and controlling the camera module to capture the second image according to the exposure parameters.
[0010] Furthermore, in some embodiments of the present invention, the step of acquiring the first image captured by the camera module includes: acquiring the original images captured by multiple camera modules respectively; and preprocessing each of the original images to obtain the first image.
[0011] Furthermore, in some embodiments of the present invention, the step of preprocessing each of the original images to obtain the first image includes: performing image analysis on each of the original images to determine the common picture of each of the original images, and determining the first image based on the image data of the common picture; or performing image analysis on each of the original images to determine the picture content of each of the original images, and selecting one of the original images as the first image based on the picture content; or performing image analysis on each of the original images to determine the difference picture of each of the original images, and splicing each of the original images based on the image data of the difference picture to determine the first image.
[0012] Furthermore, in some embodiments of the present invention, the camera module is installed in a head-mounted device. The head-mounted device is configured with a display screen and an eye tracker. The step of obtaining the user's eye movement signal to determine the coordinates of the user's gaze point in the first image includes: performing dedistortion processing on the first image according to the difference between the first resolution of the first image and the second resolution of the display screen, and displaying the obtained third image on the display screen; obtaining the user's eye movement signal via the eye tracker, and determining the coordinates of the user's first gaze point in the third image according to the eye movement signal; and performing an inverse change of the dedistortion processing on the first gaze point coordinates to determine the coordinates of the user's second gaze point in the first image.
[0013] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in the subsequent second image based on the gaze point coordinates includes: recording the exposure parameters of multiple areas in the first image in a light meter, wherein the light meter includes multiple cells, each cell corresponding to one of the areas in the first image; and performing a weighted operation on the exposure parameters in at least one cell of the light meter based on the gaze point coordinates to update the light meter.
[0014] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: determining the maximum difference in pixel grayscale values of the gaze point coordinates of the first image continuously shot by the user in multiple frames; in response to the maximum difference being less than a preset first difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates; and in response to the maximum difference being greater than or equal to the first difference threshold, maintaining the exposure parameters of each of the areas in the first image.
[0015] Further, in some embodiments of the present invention, in response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the maximum difference being less than the first difference threshold, determining a convergence speed for adjusting the exposure parameters according to the maximum difference; and adjusting the exposure parameters of each of the subsequent areas in each of the second images frame by frame according to the convergence speed.
[0016] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of each of the areas in each of the subsequent second images frame by frame according to the convergence speed includes: adjusting the exposure parameters of each of the areas in each of the second images frame by frame at a smaller first convergence speed corresponding to a larger first maximum difference; and adjusting the exposure parameters of each of the areas in each of the second images frame by frame at a larger second convergence speed corresponding to a smaller second maximum difference.
[0017] Furthermore, in some embodiments of the present invention, the step of determining the maximum difference in pixel grayscale values of the gaze point coordinates of the user in multiple frames of continuously captured first images includes: caching the pixel grayscale values of the gaze point coordinates of the multiple frames of continuously captured first images; and in response to capturing a new frame of first image, replacing the cached pixel grayscale values of the gaze point coordinates of the first frame of the first image with the pixel grayscale values of the gaze point coordinates of the new first image, and re-determining the maximum difference.
[0018] Further, in some embodiments of the present invention, in response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the re-determined maximum difference being less than the first difference threshold, determining the difference in pixel grayscale values of the user's gaze point coordinates in the last two frames of the first image; in response to the difference being less than a preset second difference threshold, maintaining the exposure parameters of each of the areas in the first image; and in response to the difference being greater than or equal to the second difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates.
[0019] Further, in some embodiments of the present invention, in response to the difference being greater than or equal to the second difference threshold, the step of adjusting the exposure parameters of each of the areas in the second image according to the gaze point coordinates includes: in response to the difference being greater than or equal to the second difference threshold and less than a preset third difference threshold, performing a weighted operation on the exposure parameters of the last frame and the last two frames of the first image to determine the exposure parameters of each of the areas in the second image; and in response to the difference being greater than or equal to the third difference threshold, adjusting the exposure parameters of each of the areas in the second image according to the exposure parameters of the gaze point area in the last frame of the first image.
[0020] Further, in some embodiments of the present invention, in response to the re-determined maximum difference being less than the first difference threshold, the step of determining the difference in pixel grayscale values of the user's gaze point coordinates in the last two frames of the first image includes: in response to the re-determined maximum difference being less than the first difference threshold, determining the gaze point area of the user in the last frame and the last two frames of the first image according to the gaze point coordinates; and in response to a judgment result that the gaze point area of the user in the last frame and the last two frames of the first image has changed, calculating the difference in pixel grayscale values of the user's gaze point coordinates in the last frame and the last two frames of the first image.
[0021] Furthermore, in some embodiments of the present invention, the step of adjusting the exposure parameters of multiple areas in the subsequent second image based on the gaze point coordinates includes: determining the pixel grayscale value of the user's gaze point coordinates in the first image; in response to the pixel grayscale value being greater than a preset grayscale value lower limit and less than a preset grayscale value upper limit, maintaining the exposure parameters of each of the areas in the first image; and in response to the pixel grayscale value being less than the grayscale value lower limit or greater than the grayscale value upper limit, adjusting the exposure parameters of each of the areas in the second image based on the gaze point coordinates.
[0022] Furthermore, in some embodiments of the present invention, the image processing method also includes the following steps: in response to the pixel grayscale value being less than the grayscale value lower limit, reducing the grayscale value upper limit and the grayscale value lower limit according to a preset relaxation step, wherein the relaxation step is less than the difference between the grayscale value upper limit and the grayscale value lower limit; and in response to the pixel grayscale value being greater than the grayscale value upper limit, increasing the grayscale value upper limit and the grayscale value lower limit according to the relaxation step.
[0023] Furthermore, in some embodiments of the present invention, the image processing method further includes the following steps: displaying the first image via a display screen; and displaying a corresponding second image according to a refresh rate of the display screen to refresh the first image.
[0024] Furthermore, an image processing device provided according to a second aspect of the present invention includes a camera module, a memory, and a processor. The memory stores computer instructions. The processor is communicatively connected to the camera module and the memory and is configured to execute the computer instructions stored in the memory to implement the image processing method described in any one of the first aspects of the present invention.
[0025] Furthermore, in some embodiments of the present invention, the image processing device further includes a display screen and an eye tracker. The display screen is configured to display the first image and / or the second image captured by the camera module. The eye tracker is configured to capture an image of a user's eyes to obtain eye movement signals from the user.
[0026] Furthermore, the computer-readable storage medium provided in the third aspect of the present invention stores computer instructions, which, when executed by a processor, implement the image processing method according to any one of the first aspects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.
[0028] FIG1 shows a schematic diagram of the architecture of an image processing apparatus according to some embodiments of the present invention.
[0029] FIG2 shows a schematic flow chart of an image processing method according to some embodiments of the present invention.
[0030] FIG3 shows a schematic diagram of gaze point coordinate conversion according to some embodiments of the present invention.
[0031] FIG4 shows a schematic diagram of a process for adjusting exposure parameters according to some embodiments of the present invention.
[0032] FIG5 shows a curve diagram showing the corresponding relationship between exposure parameter differences and weighting coefficients according to some embodiments of the present invention.
[0033] FIG6 shows a schematic diagram of a light meter according to some embodiments of the present invention.
[0034] FIG. 7 shows a schematic diagram of a process for adjusting exposure parameters according to some other embodiments of the present invention. DETAILED DESCRIPTION
[0035] The following specific embodiments illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this embodiment. On the contrary, the purpose of introducing the invention in conjunction with the embodiment is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.
[0036] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.
[0037] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood to refer to the orientations depicted in that section and the accompanying drawings. These relative terms are used solely for convenience of description and do not necessarily imply that the devices described herein must be manufactured or operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.
[0038] It will be understood that although the terms "first," "second," "third," etc. may be used herein to describe various components, regions, layers, and / or portions, these components, regions, layers, and / or portions should not be limited by these terms, and these terms are merely used to distinguish different components, regions, layers, and / or portions. Thus, a first component, region, layer, and / or portion discussed below may be referred to as a second component, region, layer, and / or portion without departing from some embodiments of the present invention.
[0039] As mentioned above, the existing automatic exposure (AE) adjustment function is generally implemented based on a pre-set fixed focus position or the recognition of finger touch signals. On the one hand, it needs to rely on the user's fingers to operate, which has the defect of being inconvenient to operate. On the other hand, it cannot be applied to the use requirements of metaverse-related devices such as head-mounted displays and VR glasses that rely on eye movement signals for control.
[0040] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image processing method, an image processing device, and a computer-readable storage medium, which can automatically adjust exposure based on eye movement signals, thereby solving the problem that extended reality display devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) cannot automatically adjust exposure through fingers, so as to meet their usage requirements.
[0041] In some non-limiting embodiments, the image processing method provided in the first aspect of the present invention can be implemented via the image processing device provided in the second aspect of the present invention. Specifically, the image processing device is configured with a memory and a processor. The memory includes, but is not limited to, the computer-readable storage medium provided in the third aspect of the present invention, having computer instructions stored thereon. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement any of the image processing methods described in the first aspect of the present invention.
[0042] Please refer to FIG. 1 for details. FIG. 1 shows a schematic diagram of the architecture of an image processing device according to some embodiments of the present invention.
[0043] As shown in FIG1 , the image processing device provided in the second aspect of the present invention may be configured with a camera module 10, a memory (not shown), a processor 20, a display screen 30, and an eye tracker 40. The image processing device may be a head-mounted device such as extended reality (XR) display glasses / mask / helmet, or a camera device such as a digital camera or smartphone.
[0044] For example, in a head-mounted device such as an MR helmet, the camera module 10 may include a left-eye real-view camera for capturing images of the user's left eye, and a right-eye real-view camera for capturing images of the user's right eye. The processor 20 may include a graphics processing unit (GPU) for acquiring original images, generating virtual images, and fused images. The display screen 30 may be integrated into the MR helmet to display real-view images, virtual images, and / or fused images of the two to the user. The eye tracker 40 may be integrated into the MR helmet to collect eye movement signals from the user's left and right eyes, respectively, as a processing basis for subsequent MR display and eye movement automatic exposure adjustment.
[0045] For another example, in a camera device such as a smartphone, the camera module 10 may include a rear camera (or cameras) located on the back of the phone, configured to capture images of the real scene behind the phone. The display screen 30 may be the main display interface located on the front of the phone, configured to display the image currently captured by the camera module 10 to the user. The eye tracker 40 may be a front-facing camera also located on the front of the phone, configured to capture eye movement signals from the user observing the display screen 30, which serve as a basis for subsequent automatic exposure adjustment based on eye movement.
[0046] The following describes the operating principles of the image processing device described above in conjunction with some examples of image processing methods. Those skilled in the art will understand that these examples of processing methods are merely non-limiting implementations of the present invention, intended to clearly demonstrate the main concepts of the present invention and provide specific solutions that facilitate implementation by the public, and are not intended to limit the full functionality or full operating methods of the processing device. Similarly, the processing device is merely a non-limiting implementation of the present invention and does not limit the execution entities or execution order of the steps in these processing methods.
[0047] Please refer to FIG. 1 and FIG. 2 . FIG. 2 shows a flowchart of an image processing method according to some embodiments of the present invention.
[0048] As shown in FIG. 1 and FIG. 2 , during the process of automatic exposure adjustment based on the eye movement signal, the processor 20 may first obtain a first image captured by the camera module 10 .
[0049] Specifically, continuing to take the above-mentioned MR helmet as an example, in the process of obtaining the first image captured by the camera module 10, the processor 20 can first obtain the left-eye original (raw) image and the right-eye original image captured by the left and right real-scene cameras of the camera module 10, respectively, and pre-process them according to a pre-configured processing strategy to integrate the original images captured by each real-scene camera into a first image.
[0050] For example, based on a preset first strategy, the processor 20 may perform image analysis on the original images captured by the left and right real-view cameras to determine a common frame of the original images, and then determine the first image based on the image data of the common frame.
[0051] For another example, based on the preset second strategy, the processor 20 can perform image analysis on the original images captured by the left and right real-view cameras to determine the picture content of each original image respectively, and select an original image that better meets the eye movement automatic exposure adjustment requirements as the first image based on the picture content.
[0052] For another example, based on a preset third strategy, the processor 20 can perform image analysis on the original images captured by the left and right real-view cameras to determine the difference between the original images. Based on the image data of the difference between the original images, the processor 20 can perform operations such as upsampling, downsampling, and rotation on the original images to unify the resolution and image angle, thereby stitching the original images together into a complete first image. Furthermore, the processor 20 can also downsample the stitched first image to meet the resolution requirements of the eye-tracking automatic exposure adjustment.
[0053] In this way, by unifying the original images captured by multiple cameras into a first image, the present invention can effectively avoid the problem of inconsistent brightness of images taken by binocular real-scene cameras, thereby improving the binocular consistency of image processing and enhancing the comfort of user observation.
[0054] Continuing with Figures 1 and 2 , after completing preprocessing of the original image and acquiring a first image, processor 20 transmits the first image to the image signal processing (ISP) pipeline of an image processing device (e.g., an MR headset), performs ISP processing on the first image, and then displays it on display screen 30. Processor 20 then obtains eye movement signals from the user observing display screen 30 via eye tracker 40 to determine the coordinates of the user's gaze point in the first image.
[0055] Please refer to FIG. 3 for details, which shows a schematic diagram of gaze point coordinate conversion according to some embodiments of the present invention.
[0056] As shown in FIG3 , in the process of determining the coordinates of the user's gaze point in the first image, the present invention can first use a dewarp algorithm to dedistort the first image based on the difference between the first resolution of the first image and the second resolution of the display screen 30, and then display the obtained third image on the display screen 30. Subsequently, the processor 20 can obtain the user's eye movement signal via the eye tracker 40 and determine the coordinates (x, y) of the user's first gaze point in the third image based on the eye movement signal. Thereafter, the processor 20 can obtain relevant parameters of the dedistortion process and, based on the parameters, perform an inverse dedistortion on the first gaze point coordinates (x, y) to determine the coordinates (x', y') of the user's second gaze point in the first image.
[0057] Please continue to refer to Figures 1 and 2. After determining the user's gaze point coordinates (x', y') in the first image, the processor 20 can input the gaze point coordinates (x', y') and the pixel grayscale values of multiple areas in the first image into an automatic exposure (AE) adjustment algorithm based on eye movement signals. The AE adjustment algorithm adjusts the exposure parameters of multiple areas in the subsequently captured second image according to the gaze point coordinates (x', y') to avoid discomfort caused by the user observing overly bright or dark areas, thereby bringing a better viewing experience to the user.
[0058] Please refer to FIG. 4 for details, which shows a schematic diagram of a process for adjusting exposure parameters according to some embodiments of the present invention.
[0059] As shown in FIG4 , during the process of adjusting the exposure parameters, the AE adjustment algorithm may first determine whether the pixel grayscale value of the user's gaze point coordinates is stable, and then adjust the exposure parameters after determining that the pixel grayscale value of the user's gaze point coordinates is stable.
[0060] In some embodiments, the AE adjustment algorithm may first determine the maximum difference X between the pixel grayscale values of the user's gaze point coordinates in the first image captured continuously over multiple frames. max -X min and compare it with the preset first difference threshold X th1 The comparison is performed to determine whether the exposure parameters of the second image captured subsequently need to be adjusted.
[0061] Specifically, in response to the maximum difference being less than a preset first difference threshold (ie, X max -X min <X th1 ), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is stable, and thus adjust the exposure parameters of each area in the second image according to the gaze point coordinates. Conversely, in response to the maximum difference being greater than or equal to the first difference threshold (i.e., X max -X min ≥X th1), the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinates is unstable, thereby maintaining the exposure parameters of each area in the first image.
[0062] In this way, the present invention can better adapt to the characteristics of rapid and continuous movement of the gaze point, by limiting the first difference threshold X th1 , to avoid unnecessary AE adjustment of multiple frames of images during the rapid and continuous movement of the gaze point, and instead start AE adjustment after the gaze point reaches the vicinity of the target position, thereby reducing the data processing load of the image processing device.
[0063] Furthermore, in order to prevent the human eye from perceiving the difference in brightness jump during the process of adjusting the exposure parameters, the AE adjustment algorithm can also preferably be adjusted based on the maximum difference X max -X min To determine the convergence speed of adjusting the exposure parameters, and adjust the exposure parameters of each area in the multiple second images captured subsequently frame by frame based on the convergence speed. Specifically, corresponding to a larger first maximum difference, the AE adjustment algorithm can use a smaller first convergence speed to adjust the exposure parameters of each area in each subsequent second image frame by frame, so as to prevent the human eye from perceiving the difference in brightness jumps during the exposure parameter adjustment process. Conversely, corresponding to a smaller second maximum difference, the AE adjustment algorithm can use a larger second convergence speed to adjust the exposure parameters of each area in each subsequent second image frame by frame, thereby improving the response rate of the AE adjustment function while ensuring observation comfort.
[0064] Furthermore, when determining the maximum difference X max -X min In the process, the AE adjustment algorithm can use an array iteration method to cache and iterate the pixel grayscale value X of the gaze point coordinates of the first image of the continuously shot n frames. i ~X i+n-1 , and update the maximum difference X of the first image of each frame accordingly max -X min Specifically, in response to acquiring a new first image, the AE adjustment algorithm may first delete the pixel grayscale value X of the first frame of the first image cached in the array. i , and the pixel grayscale value X of the first image of the remaining frames i+1 ~X i+n-1 All of them are moved forward one position, and then the pixel gray value X of the newly acquired first image is i+n Cache to the end of the array. In this way, the AE adjustment algorithm can use the pixel gray value X of the gaze point coordinate of the new first image i+n , replace the pixel gray value X of the gaze point coordinates of the first frame of the first image in the cache i , and redetermine the maximum difference in the array.
[0065] Then, in response to the re-determined maximum difference being less than the first difference threshold X th1 , the AE adjustment algorithm can determine the maximum difference from the first difference threshold X th1 When the above value falls back to the lower value, that is, the pixel grayscale value of the user's gaze point coordinates returns from an unstable state to a stable state, the user's gaze point area in the last two frames of the first image is preferably determined based on the gaze point coordinates. In response to the gaze point area moving from the first area of the last two frames of the image to the second area of the last frame of the image, the AE adjustment algorithm can determine that there is a large gaze point displacement in the last two frames of the first image, and further calculate the difference in the pixel grayscale value of the user's gaze point coordinates in the last frame and the last two frames of the first image to determine whether the exposure parameters of the subsequently captured second image need to be adjusted.
[0066] On the contrary, in response to the user's gaze point area in the last two frames of the first image remaining unchanged, the AE adjustment algorithm can determine that there is no sufficiently large gaze point displacement in the last two frames of the first image, thereby continuing to maintain the exposure parameters of each area in the first image.
[0067] In this way, the present invention can better adapt to the characteristics of frequent gaze point movement and poor stability. By limiting the change threshold of the gaze point coordinate position, it can avoid excessive AE adjustment of multiple frames of images with too small a change in the gaze point coordinates, thereby reducing the data processing load of the image processing device.
[0068] Furthermore, the AE adjustment algorithm can compare the difference X1-X2 of the pixel grayscale value of the user's gaze point coordinates in the last two frames of the first image with the preset second difference threshold X th2 The second difference threshold X is used for comparison to determine whether the exposure parameters of the second image taken subsequently need to be adjusted. th2 Can be compared with the first difference threshold X th1 has the same value, or can be the same as the first difference threshold X th1 Has different values.
[0069] Specifically, in response to the difference being less than a preset second difference threshold (ie, X1-X2<X th2 ), the AE adjustment algorithm can determine that the difference in the brightness of the gaze point in the last two frames of the first image is too small, thereby maintaining the exposure parameters of each area in the first image. On the contrary, in response to the above difference being greater than or equal to the second difference threshold (i.e., X1-X2≥X th2 ), the AE adjustment algorithm can determine that the brightness of the gaze point in the last two frames of the first image has a large difference, and thus adjust the exposure parameters of each area in the subsequently captured second image according to the above gaze point coordinates.
[0070] In this way, the present invention can better adapt to the characteristics of frequent and continuous movement of the gaze point, by limiting the second difference threshold X th2 This is to avoid excessive AE adjustment for multiple frames of images with very small differences in gaze point brightness changes, thereby reducing the data processing load of the image processing device.
[0071] Furthermore, the AE adjustment algorithm may also be preferably configured with a third difference threshold X th3 The AE adjustment algorithm can preferably be based on the third difference threshold X th3 , performing a weighted operation on the exposure parameters of at least one area in the last frame and the last two frames of the first image to adjust the exposure parameters of the corresponding areas in the subsequently captured second images.
[0072] Please refer to FIG. 4 and FIG. 5 . FIG. 5 shows a curve diagram of the corresponding relationship between exposure parameter differences and weighting coefficients provided according to some embodiments of the present invention.
[0073] As shown in FIG4 and FIG5, in the process of performing weighted operation on the exposure parameters, the AE adjustment algorithm can firstly compare the above X1-X2 difference with the second difference threshold X th2 and the third difference threshold X th3 In response to the difference being greater than or equal to the second difference threshold and less than the third difference threshold (ie, X th2 ≤X1-X2<X th3 ), the AE adjustment algorithm can query the corresponding relationship curve diagram shown in Figure 5 according to the actual difference to determine the corresponding weighting coefficient w1, and then perform weighted operation on the exposure parameters of the last frame and the last two frames of the first image according to the weighting coefficient w1 to determine the exposure parameters of each area in the second image.
[0074] Specifically, the calculation formula for performing weighted operation on the exposure parameters of the last frame and the last two frames of the first image according to the weighting coefficient w1 can be as follows: T=T n-1 (1-w1)+T n w1
[0075] Among them, T n-1 represents the exposure parameter of the first image of the last two frames, T n represents the exposure parameter of the first image of the last frame, and w1 represents the weighting coefficient.
[0076] On the contrary, in response to the difference being greater than or equal to the third difference threshold (X1-X2≥X th3 ), the AE adjustment algorithm can determine the weighting coefficient w1=1 according to the corresponding relationship curve diagram shown in FIG5 , thereby adjusting the exposure parameters of each area in the second image completely according to the exposure parameters of the gaze point area in the last frame of the first image.
[0077] Furthermore, after determining the weighting coefficient w1 of each first image involved in the weighted operation, the AE adjustment algorithm can also determine the weighting coefficient w2 of multiple areas in the first image based on the above-mentioned gaze point coordinates (x', y'), and adjust the exposure parameters of multiple areas in the subsequent second image based on the weighting coefficient w2.
[0078] Please refer to FIG6 for details, which shows a schematic diagram of a light meter provided according to some embodiments of the present invention.
[0079] As shown in Figure 6, when adjusting the exposure parameters of multiple regions in the second image based on the gaze point coordinates (x', y'), the AE adjustment algorithm can first divide the entire first image into several small regions and calculate the grayscale value of each region. The AE adjustment algorithm can then record the grayscale values of the multiple regions in the first image in multiple cells of a light meter, assign a corresponding weight w2∈[0, 1] to each cell based on the gaze point coordinates (x', y'), and then perform a weighted operation on the pixel grayscale values in at least one cell of the light meter based on the weight w2 to determine an updated light meter.
[0080] Specifically, each cell in the light meter corresponds to an area in the first image. The AE adjustment algorithm can calculate the distance from the center (a, b), (x', y'), and (c, d) of each area to the gaze point coordinates (x', y'), set a weight w2 = 1 for the area closest to the gaze point, and set a weight w2 = 0 for the area farthest from the non-gaze point.
[0081] Furthermore, in some embodiments, for a large light meter including areas of multiple different distances, the AE adjustment algorithm may also configure weights w2∈(0, 1) for multiple areas of equal distance so that they share the total weight equally (ie, 1).
[0082] Afterwards, as shown in Figures 1 and 2, the AE adjustment algorithm can calculate the grayscale value of the entire picture based on the weight w2 of each area, calculate the corresponding illumination value based on the grayscale value of the entire picture, and send the corresponding exposure parameters to the camera module 10 to control the camera module 10 to continue to shoot the subsequent second image.
[0083] Those skilled in the art will understand that the embodiment shown in FIG4 of adjusting the exposure parameters of the subsequent second image based on the difference in pixel grayscale values between multiple frames of first images is merely some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit the scope of protection of the present invention.
[0084] Optionally, in other embodiments, the AE adjustment algorithm may also achieve the effect of dynamically adjusting the exposure parameters of multiple areas in the second image captured subsequently by setting a grayscale value threshold.
[0085] Please refer to FIG. 7 for details. FIG. 7 shows a schematic flow chart of adjusting exposure parameters according to other embodiments of the present invention.
[0086] As shown in FIG7 , in response to determining the coordinates of the user's gaze point in the first image, the AE adjustment algorithm may first determine the grayscale value of the pixel at the user's gaze point in the first image and compare it with a preset grayscale lower limit Th0 and grayscale upper limit Th1. If the grayscale value of the pixel at the gaze point is between the grayscale lower limit Th0 and less than the grayscale upper limit Th1 (i.e., Th0 ≤ X ≤ Th1), the AE adjustment algorithm may determine that the grayscale value of the pixel at the user's gaze point is stable, thereby maintaining the exposure parameters of each region in the first image. Conversely, if the grayscale value of the pixel at the gaze point exceeds the grayscale lower limit Th0 or the grayscale upper limit Th1 (i.e., X < Th0 or X > Th1), the AE adjustment algorithm may determine that the grayscale value of the pixel at the user's gaze point is unstable, thereby adjusting the exposure parameters of each region in the second image based on the gaze point coordinates as described above.
[0087] Furthermore, in order to avoid too frequent jumps in exposure parameters, the AE adjustment algorithm can also dynamically adjust the grayscale value lower limit Th0 and the grayscale value upper limit ΔTh according to the jump direction of the exposure parameters to reduce the data processing load of the image processing device.
[0088] Specifically, in response to a determination that the grayscale value of the pixel at the gaze point coordinates is less than the grayscale lower limit (i.e., X < Th0), the AE adjustment algorithm can adjust the exposure parameters of each region in the second image while reducing the original grayscale lower limit Th0 and grayscale upper limit Th1 according to a preset relaxation step size ΔTh. Conversely, in response to a determination that the grayscale value of the pixel at the gaze point coordinates is greater than the grayscale upper limit (i.e., X > Th1), the AE adjustment algorithm can also increase the original grayscale lower limit Th0 and grayscale upper limit Th1 according to the relaxation step size ΔTh. Here, the relaxation step size can be less than the difference between the grayscale upper limit and the grayscale lower limit (i.e., ΔTh < Th1 - Th0) to prevent the adjusted grayscale lower limit Th0' from being greater than the original grayscale upper limit Th1, and to prevent the adjusted grayscale upper limit Th1' from being less than the original grayscale lower limit Th0. This, through relaxation, prevents the exposure parameters from jumping again in a short period of time.
[0089] In this way, the AE adjustment algorithm can also achieve the effect of dynamically adjusting the exposure parameters of multiple areas in the second image captured subsequently by setting and adjusting the grayscale value threshold.
[0090] Furthermore, as shown in Figures 1 and 2, for head-mounted devices such as augmented reality display glasses / masks / helmets with a display screen 30 and photographic devices such as digital cameras and smart phones, the processor 20 can also continue to transmit the captured second image to the display screen 30 after displaying the first image through the display screen 30, thereby controlling the display screen 30 to display the corresponding second image according to its refresh frequency to refresh the previous first image.
[0091] In summary, by determining the coordinates of the user's gaze point in the first image based on the eye movement signal, and then adjusting the exposure parameters of multiple areas in the subsequently captured second image based on the gaze point coordinates, the above-mentioned image processing method, image processing device and computer-readable storage medium provided by the present invention can not only dynamically adjust the exposure parameters of the camera following the movement of the user's line of sight to solve the problem that AR, VR, MR and other extended reality display devices cannot automatically adjust the exposure through fingers and adapt to their usage needs, but also avoid the glare phenomenon caused by the user moving the line of sight to an overly bright area, and the black screen phenomenon caused by the user moving the line of sight to an overly dark area, thereby improving the user's comfort and clarity in observing the image.
[0092] Those skilled in the art will understand that the embodiment of the image processing device shown in Figure 1, which also includes a camera module 10, a memory (not shown), a processor 20, a display screen 30 and an eye tracker 40, is merely some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concepts of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit the scope of protection of the present invention.
[0093] Optionally, in other embodiments, the image processing device does not need to be configured with the above-mentioned display screen 30 and / or eye tracker 40, but can also achieve the above-mentioned eye movement automatic exposure adjustment function by connecting to an external display screen and / or an external eye tracker.
[0094] Although the above methods are illustrated and described as a series of acts for simplicity of explanation, it is to be understood and appreciated that these methods are not limited by the order of the acts, as some acts may occur in a different order and / or concurrently with other acts from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art according to one or more embodiments.
[0095] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the foregoing description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0096] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.
[0097] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside in a user terminal as discrete components.
[0099] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image processing method, characterized in that: The following steps are involved: Acquiring a first image captured by the camera module; Acquire an eye movement signal of the user to determine the coordinates of the gaze point of the user in the first image; adjusting exposure parameters of a plurality of regions in a subsequent second image according to the gaze point coordinates; as well as According to the exposure parameters, the camera module is controlled to capture the second image.
2. The image processing method according to claim 1, characterized in that: The step of acquiring the first image captured by the camera module comprises: Acquiring original images captured by the plurality of camera modules respectively; and Preprocess each of the original images to obtain the first image.
3. The image processing method according to claim 2, characterized in that: The step of preprocessing each of the original images to obtain the first image comprises: Performing image analysis on the original images to determine a common frame of the original images, and determining the first image according to image data of the common frame; or Performing image analysis on each of the original images to determine the picture content of each of the original images, and selecting one of the original images as the first image according to the picture content; or Image analysis is performed on each of the original images to determine a difference picture of each of the original images, and each of the original images is spliced according to image data of the difference picture to determine the first image.
4. The image processing method according to claim 1, wherein: The camera module is installed in a head mounted device, and the head mounted device is provided with a display screen and an eye tracker. The step of obtaining the user's eye movement signal to determine the coordinates of the gaze point in the first image includes: performing dedistortion processing on the first image according to a difference between a first resolution of the first image and a second resolution of the display screen, and displaying the acquired third image on the display screen; Acquiring an eye movement signal of the user via the eye tracker, and determining the coordinates of a first gaze point of the user in the third image according to the eye movement signal; and An inverse change of the de-distortion process is performed on the first gaze point coordinates to determine the second gaze point coordinates of the user in the first image.
5. The image processing method according to claim 1, wherein: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Recording the grayscale values of the plurality of regions in the first image in a light meter, wherein the light meter includes a plurality of cells, each of the cells corresponding to one of the regions in the first image; and According to the gaze point coordinates, a weighted operation is performed on the grayscale value in at least one cell of the light meter to update the light meter.
6. The image processing method according to claim 1, characterized in that: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Determine a maximum difference in pixel grayscale values of the gaze point coordinates of the user in the first image continuously shot in multiple frames; In response to the maximum difference being less than a preset first difference threshold, adjusting the exposure parameter of each of the regions in the second image according to the gaze point coordinates; and In response to the maximum difference being greater than or equal to the first difference threshold, maintaining the exposure parameter of each of the regions in the first image.
7. The image processing method according to claim 6, characterized in that: In response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the maximum difference being less than the first difference threshold, determining a convergence speed for adjusting the exposure parameter according to the maximum difference; and According to the convergence speed, the exposure parameter of each of the regions in each of the subsequent second images is adjusted frame by frame.
8. The image processing method according to claim 7, characterized in that: The step of adjusting the exposure parameters of each of the regions in each of the subsequent second images frame by frame according to the convergence speed includes: corresponding to the larger first maximum difference, adjusting the exposure parameter of each of the regions in each of the second images frame by frame at a smaller first convergence speed; and Corresponding to the smaller second maximum difference, the exposure parameter of each of the regions in each of the second images is adjusted frame by frame at a larger second convergence speed.
9. The image processing method according to claim 6, characterized in that: The step of determining the maximum difference in pixel grayscale values of the gaze point coordinates of the first image continuously shot by the user in multiple frames comprises: Buffering the pixel grayscale values of the gaze point coordinates of the plurality of frames of first images captured continuously; and In response to acquiring a new first image frame, the pixel grayscale values of the gaze point coordinates of the cached first frame first image are replaced with the pixel grayscale values of the gaze point coordinates of the new first image, and the maximum difference is re-determined.
10. The image processing method according to claim 9, characterized in that: In response to the maximum difference being less than a preset first difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the re-determined maximum difference being less than the first difference threshold, determining a difference in pixel grayscale values of the gaze point coordinates of the user in the last two frames of the first image; In response to the difference being less than a preset second difference threshold, maintaining the exposure parameter of each of the regions in the first image; and In response to the difference being greater than or equal to the second difference threshold, adjusting the exposure parameter of each of the regions in the second image according to the gaze point coordinates.
11. The image processing method according to claim 10, characterized in that: In response to the difference being greater than or equal to the second difference threshold, the step of adjusting the exposure parameters of each of the regions in the second image according to the gaze point coordinates comprises: In response to the difference being greater than or equal to the second difference threshold and less than a preset third difference threshold, performing a weighted operation on the exposure parameters of the last frame and the last two frames of the first image to determine the exposure parameters of each of the areas in the second image; and In response to the difference being greater than or equal to the third difference threshold, the exposure parameters of each of the regions in the second image are adjusted according to the exposure parameters of the gaze point region in the last frame of the first image.
12. The image processing method according to claim 10, characterized in that: In response to the re-determined maximum difference being less than the first difference threshold, the step of determining the difference in pixel grayscale values of the user's gaze point coordinates in the last two frames of the first image comprises: In response to the re-determined maximum difference being less than the first difference threshold, determining the gaze point regions of the user in the last frame and the last two frames of the first image respectively according to the gaze point coordinates; and In response to a determination result that the gaze point area of the user in the last frame and the last two frames of the first image changes, a difference in pixel grayscale values of the gaze point coordinates of the user in the last frame and the last two frames of the first image is calculated.
13. The image processing method according to claim 1, characterized in that: The step of adjusting the exposure parameters of multiple areas in the subsequent second image according to the gaze point coordinates includes: Determining a pixel grayscale value of a gaze point coordinate of the user in the first image; In response to the pixel grayscale value being greater than a preset grayscale value lower limit and less than a preset grayscale value upper limit, maintaining the exposure parameter of each of the areas in the first image; and In response to the pixel grayscale value being less than the grayscale value lower limit or greater than the grayscale value upper limit, the exposure parameter of each of the regions in the second image is adjusted according to the gaze point coordinates.
14. The image processing method according to claim 13, characterized in that: The following steps are also included: In response to the pixel gray value being less than the gray value lower limit, reducing the relaxation step size according to a preset relaxation step size. The upper limit of the grayscale value and the lower limit of the grayscale value, wherein the relaxation step length is smaller than the difference between the upper limit of the grayscale value and the lower limit of the grayscale value; as well as In response to the pixel grayscale value being greater than the grayscale value upper limit, the grayscale value upper limit and the grayscale value lower limit are increased according to the relaxation step length.
15. The image processing method according to claim 1, characterized in that: The following steps are also included: displaying the first image via a display screen; and The corresponding second image is displayed according to the refresh frequency of the display screen to refresh the first image.
16. An image processing device, characterized in that: include: Camera module; a memory having computer instructions stored thereon; as well as A processor is communicatively connected to the camera module and the memory, and is configured to execute computer instructions stored in the memory to implement the image processing method according to any one of claims 1 to 15.
17. The image processing device according to claim 16, wherein: Also includes: A display screen, used to display the first image and / or the second image captured by the camera module; as well as An eye tracker is used to capture eye images of a user to obtain eye movement signals of the user.
18. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.
Citation Information
Patent Citations
Image processing method and image processing device used for mobile terminal
CN106331498A
Image processing method and device based on virtual reality, equipment and medium
CN108287678A
Photometry method based on eyeball tracking and related products
CN110414427A
Display method and device, head-mounted display equipment and storage medium
CN115883816A
Apparatuses, systems, and methods for gaze-based auto-exposure management of image frames
US20220014664A1
Cited By
Hole opening positioning method for automobile part machining
CN120318330A
Comparison analysis method and system based on AI image recognition
CN121259479A
