An image processing method, processing device and storage medium

Abstract

The application provides an image processing method, a processing device and a storage medium. The storage method comprises the following steps: acquiring a first image photographed by a camera module; acquiring an eye movement signal of a user to determine a gaze point coordinate of the user in the first image; adjusting exposure parameters of a plurality of regions in a subsequent second image according to the gaze point coordinate; and controlling the camera module to photograph the second image according to the exposure parameters. Through the above steps, the image processing method can dynamically adjust and control exposure based on the eye movement signal, thereby adapting to the application requirements of extended reality display.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method, an image processing device, and a computer readable storage medium. BACKGROUND

[0002] The existing digital cameras and camera phones are generally equipped with automatic exposure (AE) adjustment function. Users can control the camera to automatically adjust the exposure parameter in different ambient brightness scenes through the way of finger touching the focusing position, so as to collect high-quality video and images with appropriate brightness.

[0003] However, the existing automatic exposure adjustment function is generally realized based on a pre-set fixed focusing position or a recognized finger touch signal, which on the one hand needs to rely on the user's finger to operate, has the defect of troublesome operation, and on the other hand cannot be applied to the use demand of head-mounted display, VR glasses and other meta-universe related devices which rely on eye movement signals for control.

[0004] In order to overcome the above-mentioned defects existing in the prior art, the technical field urgently needs an image processing technology to solve the problem that the augmented reality (AR), virtual reality (VR), mixed reality (MR) and other extended reality display devices cannot automatically adjust the exposure through the finger, so as to adapt to their use demand. SUMMARY

[0005] The following gives a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all contemplated aspects, and is neither intended to identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description to be given later.

[0006] In order to overcome the above-mentioned defects existing in the prior art, the present application provides an image processing method, an image processing device, and a computer readable storage medium, which can automatically adjust the exposure based on the eye movement signal, so as to solve the problem that the AR, VR, MR and other extended reality display devices cannot automatically adjust the exposure through the finger, thereby adapting to their use demand.

[0007] Specifically, the image processing method according to the first aspect of the present application comprises the following steps: acquiring a first image captured by a camera module; acquiring eye movement signals of a user to determine a gaze point coordinate of the user in the first image; adjusting exposure parameters of a plurality of regions in a second image to be captured according to the gaze point coordinate; and controlling the camera module to capture the second image according to the exposure parameters.

[0008] Further, in some embodiments of the present application, the step of acquiring a first image captured by a camera module comprises: acquiring raw images captured by a plurality of camera modules respectively; and pre-processing each raw image to obtain the first image.

[0009] Further, in some embodiments of the present application, the step of pre-processing each raw image to obtain the first image comprises: image analyzing each raw image to determine a common frame of each raw image, and determining the first image according to image data of the common frame; or image analyzing each raw image to determine frame contents of each raw image, and selecting one of the raw images as the first image according to the frame contents; or image analyzing each raw image to determine different frames of each raw image, and splicing each raw image according to image data of the different frames to determine the first image.

[0010] Further, in some embodiments of the present application, the camera module is installed in a head-mounted device. The head-mounted device is configured with a display screen and an eye tracker. The step of acquiring eye movement signals of a user to determine a gaze point coordinate of the user in the first image comprises: performing de-distortion processing on the first image according to a difference between a first resolution of the first image and a second resolution of the display screen, and displaying a third image obtained to the display screen; acquiring eye movement signals of the user via the eye tracker, and determining a first gaze point coordinate of the user in the third image according to the eye movement signals; and performing inverse changes of the de-distortion processing on the first gaze point coordinate to determine a second gaze point coordinate of the user in the first image.

[0011] Further, in some embodiments of the present application, the step of adjusting exposure parameters of a plurality of regions in a second image to be captured according to the gaze point coordinate comprises: recording exposure parameters of a plurality of regions in the first image in a light meter, wherein the light meter comprises a plurality of cells, and each cell corresponds to one of the regions in the first image; and performing weighted operation on exposure parameters in at least one of the cells of the light meter according to the gaze point coordinate to update the light meter.

[0012] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the subsequent second images according to the gaze point coordinates comprises: determining a maximum difference of pixel gray values of the gaze point coordinates of the user in the plurality of continuously captured first images; in response to the maximum difference being less than a preset first difference threshold, adjusting the exposure parameters of the regions in the second images according to the gaze point coordinates; and in response to the maximum difference being greater than or equal to the first difference threshold, maintaining the exposure parameters of the regions in the first images.

[0013] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the subsequent second images according to the gaze point coordinates in response to the maximum difference being less than a preset first difference threshold comprises: in response to the maximum difference being less than the first difference threshold, determining a convergence speed of adjusting the exposure parameters according to the maximum difference; and adjusting the exposure parameters of the regions in the subsequent second images frame by frame according to the convergence speed.

[0014] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the subsequent second images frame by frame according to the convergence speed comprises: corresponding to a larger first maximum difference, adjusting the exposure parameters of the regions in the second images frame by frame with a smaller first convergence speed; and corresponding to a smaller second maximum difference, adjusting the exposure parameters of the regions in the second images frame by frame with a larger second convergence speed.

[0015] Further, in some embodiments of the present application, the step of determining the maximum difference of pixel gray values of the gaze point coordinates of the user in the plurality of continuously captured first images comprises: buffering the pixel gray values of the gaze point coordinates of the plurality of continuously captured first images; and in response to a new first image being captured, replacing the buffered pixel gray value of the gaze point coordinates of the first frame of the first images with the pixel gray value of the gaze point coordinates of the new first image, and re-determining the maximum difference.

[0016] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the second images according to the gaze point coordinates in response to the maximum difference being less than a preset first difference threshold comprises: in response to the re-determined maximum difference being less than the first difference threshold, determining a difference of pixel gray values of the gaze point coordinates of the user in the last two frames of the first images; in response to the difference being less than a preset second difference threshold, maintaining the exposure parameters of the regions in the first images; and in response to the difference being greater than or equal to the second difference threshold, adjusting the exposure parameters of the regions in the second images according to the gaze point coordinates.

[0017] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the second image according to the gaze point coordinates in response to the difference value being greater than or equal to the second difference threshold value comprises: performing a weighted operation on the exposure parameters of the last frame and the second last frame of the first image to determine the exposure parameters of the regions in the second image in response to the difference value being greater than or equal to the second difference threshold value and less than a preset third difference threshold value; and adjusting the exposure parameters of the regions in the second image according to the exposure parameters of the gaze point region in the last frame of the first image in response to the difference value being greater than or equal to the third difference threshold value.

[0018] Further, in some embodiments of the present application, the step of determining the difference value of the pixel gray values of the gaze point coordinates of the user in the last two frames of the first image in response to the re-determined maximum difference value being less than the first difference threshold value comprises: determining the gaze point regions of the user in the last frame and the second last frame of the first image according to the gaze point coordinates in response to the re-determined maximum difference value being less than the first difference threshold value; and calculating the difference value of the pixel gray values of the gaze point coordinates of the user in the last frame and the second last frame of the first image in response to a result of judging whether the gaze point regions of the user in the last frame and the second last frame of the first image change.

[0019] Further, in some embodiments of the present application, the step of adjusting the exposure parameters of the regions in the second image according to the gaze point coordinates in response to the re-determined maximum difference value being less than the first difference threshold value comprises: determining the pixel gray value of the gaze point coordinates of the user in the first image; maintaining the exposure parameters of the regions in the first image in response to the pixel gray value being greater than a preset lower gray value limit and less than a preset upper gray value limit; and adjusting the exposure parameters of the regions in the second image according to the gaze point coordinates in response to the pixel gray value being less than the lower gray value limit or greater than the upper gray value limit.

[0020] Further, in some embodiments of the present application, the image processing method further comprises the steps of: decreasing the upper gray value limit and the lower gray value limit according to a preset relaxation step length in response to the pixel gray value being less than the lower gray value limit, wherein the relaxation step length is less than a difference value between the upper gray value limit and the lower gray value limit; and increasing the upper gray value limit and the lower gray value limit according to the relaxation step length in response to the pixel gray value being greater than the upper gray value limit.

[0021] Further, in some embodiments of the present application, the image processing method further comprises the steps of: displaying the first image via a display screen; and displaying the corresponding second image according to a refresh frequency of the display screen to refresh the first image.

[0022] Further, the image processing apparatus according to the second aspect of the present application comprises a camera module, a memory and a processor. The memory has computer instructions stored thereon. The processor is communicatively connected with the camera module and the memory, and is configured to execute the computer instructions stored on the memory to implement the image processing method according to any one of the first aspect of the present application.

[0023] Further, in some embodiments of the present application, the image processing apparatus further comprises a display screen and an eye tracker. The display screen is configured to display the first image and / or the second image captured by the camera module. The eye tracker is configured to capture eye images of a user to obtain eye movement signals of the user.

[0024] Further, the computer readable storage medium according to the third aspect of the present application has computer instructions stored thereon. The computer instructions are executed by a processor to implement the image processing method according to any one of the first aspect of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above features and advantages of the present application will be better understood through the following detailed description of the embodiments of the present application in conjunction with the attached drawings. In the drawings, components are not necessarily drawn to scale and components of similar or identical function or structure can be designated with the same or similar reference numerals.

[0026] Figure 1 An architecture diagram of the image processing apparatus according to some embodiments of the present application is shown.

[0027] Figure 2 A flow diagram of the image processing method according to some embodiments of the present application is shown.

[0028] Figure 3 A diagram of the gaze point coordinate conversion according to some embodiments of the present application is shown.

[0029] Figure 4 A flow diagram of the exposure parameter adjustment according to some embodiments of the present application is shown.

[0030] Figure 5 A graph of the correspondence between the exposure parameter difference and the weighting coefficient according to some embodiments of the present application is shown.

[0031] Figure 6 A diagram of the light meter according to some embodiments of the present application is shown.

[0032] Figure 7 A flow diagram of the exposure parameter adjustment according to some other embodiments of the present application is shown. DETAILED DESCRIPTION

[0033] The present application is herein described, by way of example only, with the assistance of the accompanying drawings. As will be realized by those skilled in the art, the application is capable of other and different embodiments, and its details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the application. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

[0034] In the description of the present application, it should be noted that unless otherwise explicitly defined and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0035] In addition, "up", "down", "left", "right", "top", "bottom", "horizontal", "vertical" used in the following description should be understood as the orientation shown in the section and the related drawings. Such relative terms are only for the convenience of description, and do not mean that the device described thereby should be manufactured or operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0036] It is understood that although the terms "first", "second", "third" and the like can be used herein to describe various components, regions, layers and / or sections, these components, regions, layers and / or sections should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or sections. Therefore, the first component, region, layer and / or section discussed below can be referred to as the second component, region, layer and / or section without departing from some embodiments of the present application.

[0037] As described above, the existing automatic exposure (AE) adjustment function is generally based on a pre-set fixed focus position or a recognized finger touch signal to achieve, on the one hand, it needs to rely on the user's finger to operate, which has the defect of troublesome operation, on the other hand, it cannot be applied to the use demand of head-mounted display, VR glasses and other meta-universe related devices which rely on eye movement signal to control.

[0038] In order to overcome the above-mentioned defects in the prior art, the present application provides an image processing method, an image processing device, and a computer readable storage medium, which can automatically adjust exposure based on eye movement signals, thereby solving the problem that an extended reality display device such as Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR) cannot automatically adjust exposure by fingers to adapt to its use requirements.

[0039] In some non-limiting embodiments, the image processing method provided by the first aspect of the present application can be implemented via the image processing device provided by the second aspect of the present application. Specifically, the image processing device is configured with a memory and a processor. The memory includes but is not limited to the above-mentioned computer readable storage medium provided by the third aspect of the present application, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the image processing method described in any one of the first aspect of the present application.

[0040] Specifically, refer to Figure 1 , Figure 1 The architecture schematic diagram of the image processing device provided by some embodiments of the present application is shown.

[0041] As Figure 1 shown, the image processing device provided by the second aspect of the present application can be configured with a camera module 10, a memory (not shown), a processor 20, a display screen 30 and an eye tracker 40. Here, the image processing device can be an extended reality (XR) display glasses / mask / helmet and the like head-mounted device, or a digital camera, a smart phone and the like photographing device.

[0042] For example, in a head-mounted device such as an MR helmet, the camera module 10 can include a left-eye real scene camera for capturing a left-eye image of a user and a right-eye real scene camera for capturing a right-eye image of the user. The processor 20 can include a graphics processing unit (GPU) for obtaining an original image, generating a virtual image and fusing an image. The display screen 30 can be integrated inside the MR helmet for showing the user with a real scene image, a virtual image and / or a fused image of the two. The eye tracker 40 can be integrated inside the MR helmet for capturing eye movement signals of the left and right eyes of the user respectively as a basis for subsequent MR display and eye movement automatic exposure adjustment.

[0043] For example, in a photographing device such as a smart phone, the camera module 10 can include a rear camera (set) arranged on the back of the phone for capturing a real scene image to be photographed behind the phone. The display screen 30 can be a main display interface arranged on the front of the phone for showing the user the image currently captured by the camera module 10. The eye tracker 40 can be a front camera also arranged on the front of the phone for capturing the eye movement signal of the user observing the display screen 30 as the basis for subsequent eye movement automatic exposure adjustment.

[0044] The working principle of the image processing device described above will be described below in combination with some embodiments of image processing methods. Those skilled in the art can understand that these embodiments of processing methods are only some non-limiting embodiments provided by the present application, which are intended to clearly show the main concept of the present application and provide some specific schemes for facilitating the public to implement, but not for limiting the overall function or overall working mode of the processing device. Similarly, the processing device is also only some non-limiting embodiments provided by the present application, which does not limit the execution subject or execution order of each step in these processing methods.

[0045] Please refer to Figure 1 and Figure 2 , Figure 2 The flowchart of the image processing method provided by some embodiments of the present application is shown.

[0046] As shown in Figure 1 and Figure 2 , in the process of automatic exposure adjustment based on the eye movement signal, the processor 20 can first acquire a first image photographed by the camera module 10.

[0047] Specifically, continuing the above-mentioned MR helmet as an example, in the process of acquiring the first image photographed by the camera module 10, the processor 20 can first acquire a left eye raw image and a right eye raw image respectively photographed by the left and right eye real scene cameras of the camera module 10, and pre-process them according to a pre-configured processing strategy, so as to integrate the raw images acquired by each real scene camera into a first image.

[0048] For example, based on a pre-configured first strategy, the processor 20 can perform image analysis on each raw image acquired by the left and right eye real scene cameras to determine a common frame of each raw image, and then determine the first image according to the image data of the common frame.

[0049] For another example, based on a pre-configured second strategy, the processor 20 can perform image analysis on each raw image acquired by the left and right eye real scene cameras to respectively determine the frame content of each raw image, and then select a raw image that is more suitable for the requirement of eye movement automatic exposure adjustment as the first image according to the frame content.

[0050] For example, based on a preset third strategy, the processor 20 can perform image analysis on each raw image captured by the left and right real scene cameras to determine a difference image of each raw image, and perform operations such as up-sampling, down-sampling, rotation, and the like on each raw image to unify the resolution and image angle of each raw image according to the image data of the difference image, so as to stitch each raw image into a complete first image. Further, the processor 20 can also perform down-sampling processing on the first image obtained by stitching to make it meet the resolution requirement of the first image for eye movement automatic exposure adjustment.

[0051] In this way, by unifying the raw images captured by the plurality of cameras into a first image, the application can effectively avoid the problem of inconsistent brightness of images captured by the binocular real scene cameras, thereby improving the binocular consistency of image processing and improving the comfort of user observation.

[0052] Please continue to refer to Figure 1 and Figure 2 After completing the preprocessing operation on the raw image and obtaining the first image captured thereby, the processor 20 transmits the first image to an image signal processing (ISP) pipeline of an image processing device (for example, an MR helmet) to perform ISP processing on the first image, and then displays the first image to the display screen 30. Subsequently, the processor 20 can obtain the eye movement signal of the user observing the display screen 30 through the eye tracker 40 to determine the gaze point coordinate of the user in the first image.

[0053] For details, please refer to Figure 3 , Figure 3 a schematic diagram of gaze point coordinate conversion provided by some embodiments of the application is shown.

[0054] As shown in Figure 3 , in the process of determining the gaze point coordinate of the user in the first image, the application can first perform dewarping processing on the first image according to the difference between the first resolution of the first image and the second resolution of the display screen 30 by using a Dewarp algorithm, and display the obtained third image to the display screen 30. Subsequently, the processor 20 can obtain the eye movement signal of the user through the eye tracker 40, and determine the first gaze point coordinate (x, y) of the user in the third image according to the eye movement signal. Then, the processor 20 can obtain the relevant parameters of the dewarping processing, and perform inverse change on the first gaze point coordinate (x, y) according to the parameters to determine the second gaze point coordinate (x', y') of the user in the first image.

[0055] Please continue to refer to Figure 1 and Figure 2After determining the gaze point coordinate (x', y') of the user in the first image, the processor 20 can input the gaze point coordinate (x', y') and the pixel grayscale values of the plurality of regions in the first image into an automatic exposure (AE) adjustment algorithm based on the eye movement signal, and adjust the exposure parameters of the plurality of regions in a second image to be subsequently captured according to the gaze point coordinate (x', y') via the AE adjustment algorithm, so as to avoid the discomfort caused by the user observing overly bright or overly dark regions, thereby providing a better viewing experience for the user.

[0056] For a better understanding of the present application, reference will be made to the following detailed description of the application. Figure 4 Figure 4 A flowchart of adjusting exposure parameters is shown according to some embodiments of the present application.

[0057] As shown in Figure 4 , in the process of adjusting the exposure parameters, the AE adjustment algorithm can first determine whether the pixel grayscale value of the user's gaze point coordinate is stable, and only after determining that the pixel grayscale value of the user's gaze point coordinate is stable, the adjustment of the exposure parameters is performed.

[0058] In some embodiments, the AE adjustment algorithm can first determine the maximum difference X max -X min in the pixel grayscale values of the gaze point coordinates of the user in a plurality of continuously captured first images, and compare it with a preset first difference threshold X th1 to determine whether the exposure parameters of a second image to be subsequently captured need to be adjusted.

[0059] Specifically, in response to the maximum difference being less than the preset first difference threshold (i.e., X max -X min <X th1 , the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinate is stable, and thus adjust the exposure parameters of each region in the second image according to the gaze point coordinate. Conversely, in response to the maximum difference being greater than or equal to the first difference threshold (i.e., X max -X min ≥X th1 , the AE adjustment algorithm can determine that the pixel grayscale value of the user's gaze point coordinate is not stable, and thus maintain the exposure parameters of each region in the first image.

[0060] In this way, the present application can better adapt to the characteristics of rapid and continuous movement of the gaze point, and by limiting the above-mentioned first difference threshold X th1 , avoid making unnecessary AE adjustments on a plurality of images during the rapid and continuous movement of the gaze point, and instead start making AE adjustments after the gaze point reaches the vicinity of the target position, thereby reducing the data processing load of the image processing device.

[0061] ​Further, in order to avoid the human eye from perceiving the difference in luminance jump during the adjustment of the exposure parameter, the AE adjustment algorithm can further determine a convergence speed of the adjustment of the exposure parameter according to the maximum difference X max -X min , and adjust the exposure parameter of each region in each of the subsequent second images frame by frame according to the convergence speed. Specifically, corresponding to a larger first maximum difference, the AE adjustment algorithm can adjust the exposure parameter of each region in each of the subsequent second images frame by frame using a smaller first convergence speed, so as to avoid the human eye from perceiving the difference in luminance jump during the adjustment of the exposure parameter. Conversely, corresponding to a smaller second maximum difference, the AE adjustment algorithm can adjust the exposure parameter of each region in each of the subsequent second images frame by frame using a larger second convergence speed, so as to improve the response rate of the AE adjustment function while ensuring the viewing comfort.

[0062] Further, in the process of determining the maximum difference X max -X min , the AE adjustment algorithm can use an array iteration manner to cache and iterate the pixel gray values X i ~X i+n-1 of the gaze point coordinates of the n frames of the first images successively captured, and update the maximum difference X max -X min of each frame of the first images according to the same. Specifically, in response to the acquisition of a new frame of the first images, the AE adjustment algorithm can first delete the pixel gray value X i of the first frame of the first images cached in the array, and sequentially shift the pixel gray values X i+1 ~X i+n-1 of the remaining frames of the first images by one position, and then cache the pixel gray value X i+n of the newly acquired first image to the last position of the array. In this way, the AE adjustment algorithm can replace the pixel gray value X i+n of the gaze point coordinates of the first frame of the first images cached with the pixel gray value X i of the gaze point coordinates of the new first image, and re-determine the maximum difference in the array.

[0063] Subsequently, in response to the re-determined maximum difference being less than the first difference threshold X th1 , the AE adjustment algorithm can determine that the maximum difference is less than the first difference threshold X th1The above falls back to the pixel gray value of the user's gaze point coordinates from the unstable state to the stable state, thereby preferably determining the user's gaze point area in the last two frames of the first image according to the gaze point coordinates. In response to the gaze point area moving from the first area of the last two frames of the image to the second area of the last frame of the image, the AE adjustment algorithm can determine that there is a large gaze point displacement in the last two frames of the first image, thereby further calculating the difference between the pixel gray values of the gaze point coordinates of the user in the last two frames of the first image to determine whether the exposure parameters of the subsequently photographed second image need to be adjusted.

[0064] Conversely, in response to the user's gaze point area in the last two frames of the first image being unchanged, the AE adjustment algorithm can determine that there is not a large enough gaze point displacement in the last two frames of the first image, thereby continuing to maintain the exposure parameters of each area in the first image.

[0065] In this way, the present application can better adapt to the characteristics of frequent gaze point movement and poor stability, avoid excessive AE adjustment of multiple images with too small gaze point coordinate changes by limiting the change threshold of the gaze point coordinate position, and thereby reduce the data processing load of the image processing device.

[0066] Furthermore, the AE adjustment algorithm can compare the difference X1-X2 between the pixel gray values of the gaze point coordinates of the user in the last two frames of the first image with a preset second difference threshold X th2 to determine whether the exposure parameters of the subsequently photographed second image need to be adjusted. Here, the second difference threshold X th2 may have the same value as the first difference threshold X th1 , or have a different value from the first difference threshold X th1 .

[0067] Specifically, in response to the above difference being less than the preset second difference threshold (i.e., X1-X2 th2 ), the AE adjustment algorithm can determine that the change difference of the gaze point luminance of the last two frames of the first image is too small, thereby maintaining the exposure parameters of each area in the first image. Conversely, in response to the above difference being greater than or equal to the second difference threshold (i.e., X1-X2 th2 ), the AE adjustment algorithm can determine that the change difference of the gaze point luminance of the last two frames of the first image is large, thereby adjusting the exposure parameters of each area in the subsequently photographed second image according to the above gaze point coordinates.

[0068] In this way, the present application can better adapt to the characteristics of frequent and continuous movement of the gaze point, avoid excessive AE adjustment of multiple images with too small change difference of the gaze point luminance by limiting the second difference threshold X th2 , and thereby reduce the data processing load of the image processing device.

[0069] Further, the AE adjustment algorithm can be further configured with a third difference threshold X th3 . The AE adjustment algorithm can preferably perform a weighted operation on the exposure parameters of at least one region in the last frame and the second-to-last frame first images based on the third difference threshold X th3 to adjust the exposure parameters of the corresponding regions in the second image.

[0070] Please refer to Figure 4 and Figure 5 , Figure 5 for a curve diagram showing the correspondence between the exposure parameter difference and the weighting coefficient according to some embodiments of the present application.

[0071] As shown in Figure 4 and Figure 5 , in the process of performing the weighted operation on the exposure parameters, the AE adjustment algorithm can first compare the above-mentioned X th2 -X th3 difference with a second difference threshold X th2 and a third difference threshold X th3 . In response to the difference being greater than or equal to the second difference threshold and less than the third difference threshold (i.e. X th2 ≤X th3 ), the AE adjustment algorithm can determine the corresponding weighting coefficient w Figure 5 according to the curve diagram shown in Figure 5 , and then perform the weighted operation on the exposure parameters of the last frame and the second-to-last frame first images according to the weighting coefficient w Figure 5 to determine the exposure parameters of the regions in the second image.

[0072] Specifically, the calculation formula for performing the weighted operation on the exposure parameters of the last frame and the second-to-last frame first images according to the weighting coefficient w Figure 5 may be as follows:

[0073] T n-1 = T n (1-w Figure 5 )+T n-1 w Figure 5

[0074] wherein T n-1 represents the exposure parameter of the second-to-last frame first image, T n represents the exposure parameter of the last frame first image, and w Figure 5 represents the weighting coefficient.

[0075] Conversely, in response to the difference being greater than or equal to the third difference threshold (X th3 ), the AE adjustment algorithm can determine the weighting coefficient w Figure 5 = 1 according to the curve diagram shown in Figure 5 , so as to adjust the exposure parameters of the regions in the second image completely according to the exposure parameters of the gaze point region in the last frame first image.

[0076] Furthermore, after determining the weighting coefficient w1 of each first image participating in the weighting calculation, the AE adjustment algorithm can also determine the weighting coefficient w2 of multiple regions in the first image based on the above-mentioned gaze point coordinates (x', y'), and adjust the exposure parameters of multiple regions in the subsequent second image based on the weighting coefficient w2.

[0077] Please refer to the details. Figure 6 , Figure 6 A schematic diagram of a photometer provided according to some embodiments of the present invention is shown.

[0078] like Figure 6 As shown, in the process of adjusting the exposure parameters of multiple regions in the second image according to the gaze point coordinates (x', y'), the AE adjustment algorithm can first divide the entire image of the first image into several small regions and count the grayscale value of each region. Then, the AE adjustment algorithm can record the grayscale values ​​of multiple regions in the first image into multiple cells of the metering table, assign a corresponding weight w2∈[0,1] to each cell according to the gaze point coordinates (x', y'), and then perform a weighted calculation on the pixel grayscale values ​​in at least one cell of the metering table according to the weight w2 to determine the updated metering table.

[0079] Specifically, each cell in the light meter corresponds to a region in the first image. The AE adjustment algorithm can calculate the distance from the center (a,b), (x',y'), and (c,d) of each region to the gaze point coordinates (x',y'), set a weight w2=1 for the region closest to the gaze point, and set a weight w2=0 for the region farthest from the gaze point.

[0080] Furthermore, in some embodiments, for a large photometer containing multiple regions at different distances, the AE adjustment algorithm can also assign weights w2∈(0,1) to multiple regions at equal distances so that they share the total weight (i.e., 1).

[0081] After that, such as Figure 1 and Figure 2 As shown, the AE adjustment algorithm can calculate the grayscale value of the entire image based on the weight w2 of each region, calculate the corresponding illuminance value based on the grayscale value of the entire image, and send the corresponding exposure parameters to the camera module 10 to control the camera module 10 to continue shooting the subsequent second image.

[0082] Those skilled in the art will understand that Figure 4 The embodiments shown, which adjust the exposure parameters of subsequent second images based on the differences in pixel grayscale values ​​between multiple frames of the first images, are merely some non-limiting implementations provided by the present invention. They are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are easy for the public to implement, rather than to limit the scope of protection of the present invention.

[0083] Optionally, in some other embodiments, the AE adjustment algorithm can also achieve the effect of dynamically adjusting the exposure parameters of the multiple regions in the second image to be captured subsequently by setting the threshold of the gray value.

[0084] For details, please refer to Figure 7 , Figure 7 A flowchart of adjusting the exposure parameters according to some other embodiments of the present application is shown.

[0085] As shown in Figure 7 , in response to determining the gaze point coordinate of the user in the first image, the AE adjustment algorithm can first determine the pixel gray value of the gaze point coordinate of the user in the first image and compare it with the preset lower limit Th0 and upper limit Th1 of the gray value. In response to the judgment result that the pixel gray value of the gaze point coordinate is between the lower limit Th0 and the upper limit Th1 of the gray value (i.e. Th0≤X≤Th1), the AE adjustment algorithm can determine that the pixel gray value of the gaze point coordinate of the user is stable, so as to maintain the exposure parameters of the regions in the first image. Conversely, in response to the judgment result that the pixel gray value of the gaze point coordinate exceeds the lower limit Th0 or the upper limit Th1 of the gray value (i.e. X<Th0 or X>Th1), the AE adjustment algorithm can determine that the pixel gray value of the gaze point coordinate of the user is unstable, so as to adjust the exposure parameters of the regions in the second image according to the gaze point coordinate as described above.

[0086] Further, in order to avoid the exposure parameters from jumping too frequently, the AE adjustment algorithm can also dynamically adjust the above-mentioned lower limit Th0 and upper limit ΔTh of the gray value according to the direction of the jump of the exposure parameters, so as to reduce the data processing load of the image processing device.

[0087] Specifically, in response to the judgment result that the pixel gray value of the gaze point coordinate is less than the lower limit of the gray value (i.e. X<Th0), the AE adjustment algorithm can decrease the original lower limit Th0 and upper limit Th1 of the gray value according to the preset relaxation step ΔTh while adjusting the exposure parameters of the regions in the second image. Conversely, in response to the judgment result that the pixel gray value of the gaze point coordinate is greater than the upper limit of the gray value (i.e. X>Th1), the AE adjustment algorithm can also increase the original lower limit Th0 and upper limit Th1 of the gray value according to the relaxation step ΔTh. Here, the relaxation step can be less than the difference between the upper limit and the lower limit of the gray value (i.e. ΔTh<Th1-Th0), so as to avoid the adjusted lower limit Th0' of the gray value being greater than the original upper limit Th1 of the gray value and avoid the adjusted upper limit Th1' of the gray value being less than the original lower limit Th0 of the gray value, thereby avoiding the re-jump of the exposure parameters in a short time through the relaxation change.

[0088] In this way, the AE adjustment algorithm can achieve the effect of dynamically adjusting the exposure parameters of the multiple regions in the second image to be captured subsequently by setting and adjusting the gray value threshold.

[0089] Further, as shown in Figure 1 and Figure 2 , for the head-mounted device such as the extended reality display glasses / mask / helmet with the display screen 30 and the digital camera, the smart phone and other photographing devices, the processor 20 can continue to transmit the captured second image to the display screen 30 after displaying the first image via the display screen 30, so as to control the display screen 30 to display the corresponding second image according to the refresh frequency thereof to refresh the previous first image.

[0090] In summary, by determining the gaze point coordinates of the user in the first image according to the eye movement signal, and then adjusting the exposure parameters of the multiple regions in the second image to be captured subsequently according to the gaze point coordinates, the image processing method, the image processing device and the computer readable storage medium provided by the present application not only can dynamically adjust the exposure parameters of the camera to follow the movement of the user's line of sight, so as to solve the problem that the extended reality display device such as AR, VR and MR cannot be automatically adjusted by the finger, and adapt to the use demand, but also can avoid the glare phenomenon when the user moves the line of sight to the over-bright region, and the black screen phenomenon when the user moves the line of sight to the over-dark region, so as to improve the comfort and clarity of the user observing the image.

[0091] Those skilled in the art can understand that Figure 1 the embodiment of the image processing device shown in

[0092] Optionally, in other embodiments, the display screen 30 and / or the eye tracker 40 can also not be necessarily configured in the image processing device, but by connecting an external display screen and / or an external eye tracker, the function of the above-mentioned eye movement automatic exposure adjustment can also be achieved.

[0093] Although the above methods are illustrated and described as a series of acts for the sake of simplicity, it should be understood and appreciated that the methods are not limited by the order of acts, as some acts can occur in different orders and / or concurrently with other acts according to one or more embodiments. Not all steps are necessarily present in every implementation of the methods, and not all of the methods make use of all of the steps presented.

[0094] Those skilled in the art will appreciate that information, signals, and data can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0095] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0096] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0097] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0098] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, Includes the following steps: Multiple frames of the first image are captured continuously via the camera module; Acquire the user's eye movement signals to determine the coordinates of their gaze point in each of the first images; Determine the maximum difference in pixel grayscale values ​​of the gaze point coordinates of the first image captured by the user in multiple consecutive frames; In response to the maximum difference being less than a preset first difference threshold, the convergence speed of adjusting the exposure parameters is determined based on the maximum difference, and the exposure parameters of multiple regions in subsequent second images are adjusted frame by frame based on the gaze point coordinates and the convergence speed. In response to the maximum difference being greater than or equal to the first difference threshold, the exposure parameters of each region in the first image are maintained. as well as Based on the exposure parameters, the camera module is controlled to capture the second image.

2. The image processing method as described in claim 1, characterized in that, The step of acquiring the first image captured by the camera module includes: The original images captured by each of the multiple camera modules are obtained; and The original images are preprocessed to obtain the first image.

3. The image processing method as described in claim 2, characterized in that, The step of preprocessing each of the original images to obtain the first image includes: Image analysis is performed on each of the original images to determine a common scene among the original images, and the first image is determined based on the image data of the common scene; or Image analysis is performed on each of the original images to determine the content of each original image, and one of the original images is selected as the first image based on the content; or Image analysis is performed on each of the original images to determine the differences between the original images, and the original images are stitched together based on the image data of the differences to determine the first image.

4. The image processing method as described in claim 1, characterized in that, The camera module is mounted on a head-mounted device, which includes a display screen and an eye tracker. The step of acquiring the user's eye movement signals to determine the coordinates of their gaze point in the first image includes: Based on the difference between the first resolution of the first image and the second resolution of the display screen, the first image is subjected to distortion correction processing, and the obtained third image is displayed on the display screen; The eye movement signals of the user are acquired via the eye tracker, and the coordinates of the user's first gaze point in the third image are determined based on the eye movement signals; and The first gaze point coordinates are subjected to the inverse transformation of the distortion removal process to determine the user's second gaze point coordinates in the first image.

5. The image processing method as described in claim 1, characterized in that, The step of adjusting the exposure parameters of multiple regions in the subsequent second image based on the gaze point coordinates includes: The grayscale values ​​of multiple regions in the first image are recorded in a light meter, wherein the light meter includes multiple cells, each cell corresponding to one of the regions in the first image; and Based on the gaze point coordinates, a weighted calculation is performed on the gray values ​​in at least one cell of the light meter to update the light meter.

6. The image processing method as described in claim 1, characterized in that, The step of adjusting the exposure parameters of each region in subsequent second images frame by frame based on the gaze point coordinates and the convergence speed includes: Corresponding to the first maximum difference, the exposure parameters of each region in each of the second images are adjusted frame by frame at a first convergence speed; and For a second maximum difference that is less than the first maximum difference, the exposure parameters of each region in each of the second images are adjusted frame by frame at a second convergence speed that is greater than the first convergence speed.

7. The image processing method as described in claim 1, characterized in that, It also includes the following steps: The first image is displayed on the screen; and The second image is displayed according to the refresh rate of the display screen to refresh the first image.

8. An image processing method, characterized in that, Includes the following steps: Multiple frames of the first image are captured continuously via the camera module; Acquire the user's eye movement signals to determine the coordinates of their gaze point in each of the first images; Cache the pixel grayscale values ​​of the gaze point coordinates of the first image captured in multiple consecutive frames; In response to the acquisition of a new first image frame, the pixel grayscale value of the gaze point coordinate of the cached first image frame is replaced with the pixel grayscale value of the gaze point coordinate of the new first image frame, so as to redetermine the maximum difference of the pixel grayscale values. In response to the re-determined maximum difference being less than a preset first difference threshold, the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last two frames of the first image is determined. In response to the difference being greater than or equal to a preset second difference threshold and less than a preset third difference threshold, the exposure parameters of the last frame and the last two frames of the first image are weighted to determine the exposure parameters of multiple regions in the subsequent second image. In response to the difference being greater than or equal to the third difference threshold, the exposure parameters of each region in the second image are adjusted according to the exposure parameters of the gaze point region in the last frame of the first image. In response to the maximum difference being greater than or equal to the first difference threshold, the exposure parameters of each region in the first image are maintained. as well as Based on the exposure parameters, the camera module is controlled to capture the second image.

9. The image processing method as described in claim 8, characterized in that, It also includes the following steps: In response to the difference being less than the second difference threshold, the exposure parameters of each region in the first image are maintained.

10. The image processing method as described in claim 8, characterized in that, The step of determining the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last two frames of the first image in response to the re-determined maximum difference being less than the first difference threshold includes: In response to the re-determined maximum difference being less than the first difference threshold, the gaze point regions of the user in the last frame and the last two frames of the first image are determined based on the gaze point coordinates; and In response to the determination that the user's gaze point region changes in the last frame and the first image of the last two frames, the difference in pixel grayscale values ​​of the user's gaze point coordinates in the last frame and the first image of the last two frames is calculated.

11. An image processing method, characterized in that, Includes the following steps: The first image captured by the camera module is obtained; Acquire the user's eye movement signals to determine their gaze coordinates in the first image: Determine the pixel grayscale value of the user's gaze point coordinates in the first image; In response to the pixel grayscale value being greater than a preset lower limit and less than a preset upper limit, the exposure parameters of each region in the first image are maintained. In response to a pixel grayscale value being less than the lower grayscale value limit or greater than the upper grayscale value limit, the exposure parameters of multiple regions in the subsequent second image are adjusted according to the gaze point coordinates, and the lower grayscale value limit and the upper grayscale value limit are dynamically adjusted according to the direction of the exposure parameter jumps, wherein the dynamic adjustment is achieved through a preset relaxation step size; and Based on the exposure parameters, the camera module is controlled to capture the second image.

12. The image processing method as described in claim 11, characterized in that, The step of dynamically adjusting the lower limit and the upper limit of the grayscale value according to the direction of the change in the exposure parameters includes: In response to the pixel grayscale value being less than the lower grayscale value limit, the upper grayscale value limit and the lower grayscale value limit are decreased according to the relaxation step size, wherein the relaxation step size is less than the difference between the upper grayscale value limit and the lower grayscale value limit; and In response to the pixel grayscale value being greater than the upper limit of the grayscale value, the upper limit of the grayscale value and the lower limit of the grayscale value are increased according to the relaxation step size.

13. An image processing apparatus, characterized in that, include: Camera module; Memory, on which computer instructions are stored; as well as The processor is communicatively connected to the camera module and the memory, and is configured to execute computer instructions stored in the memory to implement the image processing method as described in any one of claims 1 to 12.

14. The image processing apparatus as claimed in claim 13, characterized in that, Also includes: A display screen for displaying a first image and / or a second image captured by the camera module; as well as An eye tracker is used to capture images of a user's eyes in order to obtain the user's eye movement signals.

15. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the image processing method as described in any one of claims 1 to 12 is implemented.

Images