Method and electronic device for blur reduction of occlusion mask for virtual image in augmented reality glasses

Abstract

The present disclosure provides methods and electronic devices, for blur reduction of an occlusion mask. A method includes determining position, size, shape and amount of light blocked by the occlusion mask, wherein the position, size and shape of the occlusion mask is determined based on position, size and shape of a virtual image, and the amount of light blocked by the occlusion mask is determined based on surrounding brightness, obtaining, by an optical system, an occlusion mask image in an image space and changing transmittance of electroactive panel to reduce remaining blur by transmittance variation.

Description

METHOD AND ELECTRONIC DEVICE FOR BLUR REDUCTION OF OCCLUSION MASK FOR VIRTUAL IMAGE IN AUGMENTED REALITY GLASSES

[0001] The present disclosure relates to augmented reality (AR) glasses, and more specifically to displaying undistorted virtual image.

[0002] Wearable augmented reality (AR) glasses are a personal device that can be used as a source of video information (image) projected directly into user's eye in the form of a virtual image augmenting the real world visibletothe user. The prior art problem is that with the augmented reality glasses, the user sees virtual image as pale, transparent, and fuzzy against bright image of real world. To improve virtual image visibility against the bright real world image, occlusion masks are used , which represent a darkening background on which virtual images are overlaid.

[0003] Since the occlusion mask is placed in front of the user's eye, i.e. the distance from the occlusion mask to the eye is small, the eye cannot focus, due to anatomical features, on the virtual image overlaid on the darkening background (occlusion mask), so the occlusion mask will be visible, from under the virtual image, blurry at the edges. The blur can be eliminated by moving the occlusion mask away from the eye, but in such event the device that forms the occlusion mask will be large i.e. its use in augmented reality glasses will become impossible.

[0004] There is a need to display an occlusion mask used in augmented reality glasses such that the virtual image can be clearly seen and contrasting against the bright real world image owing to the absence of blur of the occlusion mask on which the virtual image is overlaid.

[0005] According to an embodiment of the disclosure, an electronic device for blur reduction of an occlusion mask is disclosed herein. The electronic device may include an optical system, memory configured to store instructions, and one or more processors. The instructions, when executed by the one or more processors, may cause the electronic device to determine position, size, shape and amount of light blocked by the occlusion mask, wherein the position, size and shape of the occlusion mask is determined based on position, size and shape of a virtual image. The amount of light blocked by the occlusion mask is determined based on surrounding brightness. The instructions, when executed by the one or more processors, may cause the electronic device to obtain, by the optical system, an occlusion mask image in an image space. The instructions, when executed by the one or more processors, may cause the electronic device to change transmittance of electroactive panel to reduce remaining blur by transmittance variation.

[0006] According to an embodiment of the disclosure, a method for blur reduction of an occlusion mask is disclosed herein. The method may include determining position, size, shape and amount of light blocked by the occlusion mask. The position, size and shape of the occlusion mask may be determined based on position, size and shape of a virtual image, and the amount of light blocked by the occlusion mask is determined based on surrounding brightness. The method may include obtaining, by an optical system, an occlusion mask image in an image space. The method may include changing transmittance of electroactive panel to reduce remaining blur by transmittance variation.

[0007] According to an embodiment of the disclosure, a computer-readable medium is disclosed herein. The computer-readable medium is configured to store instructions which, when executed by at least one processor of an electronic device, cause the electronic device to perform the method.

[0008] Fig. 1 illustrates an embodiment of a comparison of the results of using related solutions and the present invention.

[0009] Fig. 2 is a schematic view illustrating an extended occlusion mask image and the dependence of normalized distribution of light intensity on point position in image space, which defines this image.

[0010] Fig. 3 illustrates an embodiment of the extended occlusion mask area on electroactive panel.

[0011] Fig. 4 illustrates an embodiment of reduction of image nonuniformity in the central part of occlusion mask.

[0012] Fig. 5 shows a schematic structure diagram of an electronic device to which an embodiment of the present disclosure is applied.

[0013] Fig. 6A and Fig. 6B illustrates a usability and use cases of the present disclosure.

[0014] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it is to be apparent to those skilled in the art that these concepts may be practiced without these specific details. In an instance, well known structures and components are shown in block diagram form in order to avoid obscuring such concepts. In the descriptions that follow, like parts are marked throughout the specification and drawings with the same numerals, respectively.

[0015] The following description provides examples, and is not limiting of the scope, applicability, or embodiments set forth in the claims. Changes may be made in the function and / or arrangement of elements discussed without departing from the scope of the present disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, and / or combined. Alternatively or additionally, features described with reference to some examples may be combined in other examples.

[0016] Various aspects and / or features may be presented in terms of systems that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, and the like and / or may not include all of the devices, components, modules, and the like discussed in connection with the figures. A combination of these approaches may also be used.

[0017] Expressions such as "at least one of," when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list. For example, the expression, "at least one of a, b, and c," should be understood as including only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or any variations of the aforementioned examples.

[0018] While such terms as "first," "second," etc., may be used to describe various elements, such elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

[0019] The term "module" or "component" is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

[0020] As a general introduction to the subject matter described in more detail below, aspects described herein are directed towards electronic devices, methods, systems, and computer-readable mediums for image blur reduction of an occlusion mask used for a virtual image displayed against a real world image with augmented reality glasses. The present disclosure used in augmented reality glasses enables the user to see a virtual image contrast and bright, and clearly distinguishable against the bright real world.

[0021] Terms used within the scope of this disclosure mean the following:

[0022] Electroactive panel is a screen consisting of elements, where each element of the electroactive panel can be either open and transmit surrounding light, or closed and do not transmit surrounding light, also each element can partially transmit surrounding light. Light transmittance of each element is controlled by signals (e.g. from a controller) to each element. In an embodiment, electroactive panel may be SLM, LCD, but not limited thereto. Electroactive panel may be implemented on tunable phase gratings.

[0023] Occlusion mask is an assembly of elements on the electroactive panel for forming an occlusion area, i.e. an area whose bounds coincide with bounds of a virtual image; moreover, the electroactive panel elements in the occlusion area can block, transmit, or partially transmit surrounding light. Virtual image may be overlaid on the occlusion mask.

[0024] The present invention provides, arranged along an optical axis:

[0025] an optical system for forming a visible image in an image space, the optical system comprising an image receiver, e.g. eye or a camera;

[0026] an entrance pupil of the optical system;

[0027] AR glasses whose screen is parallel to the entrance pupil plane, where a virtual image having a specified position on the screen, a specified shape, and a specified size is disposed on the AR glasses screen;

[0028] an electroactive panel parallel to the AR glasses screen, the electroactive panel being disposed at a known distance from the plane of the optical system entrance pupil and at a known distance from the plane of the AR glasses screen on which the virtual image is located;

[0029] real world.

[0030] Electroactive panel can be secured to the AR glasses screen with an optically transparent adhesive, moreover, the AR glasses and the electroactive panel can have a common frame. An occlusion mask having the same shape as the virtual image is formed on the electroactive panel.

[0031] Fig. 1 shows a comparison of the results of using of related solutions and the present invention. Fig. 1 depicts:

[0032] object space 110, where virtual image 114 is overlaid on real world image 112;

[0033] area 120, where electroactive panel 140, on which an occlusion mask is formed, and entrance pupil 150 of optical system 160 are disposed;

[0034] image space 130, where image is displayed, e.g. eye retina.

[0035] In an embodiment, the optical system 130 may be user's eye or any other suitable optical system, e.g. a camera may be used as the optical system 5 in devices for people with photophobia.

[0036] If no occlusion mask is used to form a virtual image, the user will see virtual image semi-transparent against a bright real world image 132.

[0037] Optical performance of any optical system may be limited by optical aberrations, diffraction, and spread, so display of the occlusion mask image in the image space, including the effects of aberrations, diffraction and spread of the optical system, may be blurry. In an embodiment, virtual image may be formed in a blurred halo, i.e. the virtual image space is surrounded by a blur area, as shown in 134. This effect cannot be completely eliminated, but the present disclosure may reduce it.

[0038] The present disclosure may provide a method, an electronic device, a computer medium for forming an occlusion mask by which a contrast and bright image of a virtual object can be formed in 136.

[0039] In an embodiment, for convenience, the position of the virtual image, and therefore the position of the occlusion mask, may be considered in the angular field area with center in the center of the entrance pupil of the optical system. Therefore, the position of a point on the electroactive panel and corresponding position of the same point in the image space may be initially known.

[0040] Point Spread Function (PSF), e.g. the dependence of light distribution on coordinates of a point in the image plane, may be determined for each optical system. PSF may take into account the aforementioned aberrations, diffraction, and spread of the optical system. Point spread function of an optical system may be determined, for example, by optical system simulation software, such as Zemax, approximation models (e.g. SMART DIMMING SUNGLASSES FOR PHOTOPHOBIA USING SPATIAL LIGHT MODULATOR, April 17, 2023, Xiaodan Hu, et al. Shanghai Jiao Tong University; Nara Institute of Science and Technology Smart Dimming Sunglasses for Photophobia Using Spatial Light Modulator, DeepAI, or article by Konyukhov A.L. Determining Point Spread Function from Characteristic Fragments of Images, Konyukhov A.L., Kostevich A.G., Kuryachiy M.I., ELECTRONICS, MEASURING EQUIPMENT, RADIO ENGINEERING AND COMMUNICATIONS, Proceedings of TUSUR, No.2(26), Part 1, December 2012, but not limited thereto). In an embodiment, PSF of an optical system may depend on the distance of the optical system to the object perceived by the optical system. In an embodiment, PSF may be obtained from programs for optical system design. In an embodiment, PSF may be obtained from the blurry camera image.

[0041] In an embodiment, the entire area of the electroactive panel, and therefore the occlusion mask on the electroactive panel, may be defined by the dependence of surrounding light transmittance at each point on the position (coordinates) of the point on the electroactive panel (hereinafter referred to as transmittance dependence on the point position on the electroactive panel). Image of an electroactive panel with an occlusion mask in the image space perceived by the optical system may be defined by the dependence of normalized distribution of light intensity at each point on the position of the point in the image space (hereinafter referred to as dependence of normalized distribution of light intensity on the point position in the image space). Point positions in the image space may be easily converted to point positions on a real electroactive panel and vice versa.

[0042] Occlusion mask may correspond to the shape and position of virtual image, and its transmittance may be automatically calculated and re-adjusted each time the surrounding light changes. Therefore, the amount of light to be blocked may be known initially and depend on the surrounding light brightness (e.g. brightness level passing through the virtual image). If an occlusion mask contains only fully blocked elements on the electroactive panel, image blur of the occlusion mask may not be eliminated because there is no variation for any change in the transmittance of elements to observe the condition of constant amount of blocked light since all light is blocked. Therefore, the occlusion mask is implemented by partially darkening the electroactive panel elements.

[0043] In an embodiment, to produce an extended occlusion mask, which electroactive panel elements to be blocked accordingto the specified shape and size of the virtual image may be determined. Image of the extended occlusion mask may include an area corresponding to the virtual image size, and a blur area calculated with the PSF.

[0044] Initially known and specified parameters may include: optical system magnification, distance from entrance pupil 150 of optical system 160 to electroactive panel 140, focal length of optical system 160, diameter of entrance pupil 150 (for eye - diameter of pupil), shape of virtual image, size of virtual image, position of virtual image in the optical system field of view, amount of light (brightness) to be blocked by occlusion mask, which may depend on surrounding brightness, Point Spread Function (PSF) of optical system, which depend on the specified distance from the optical system entrance pupil to the electroactive panel.

[0045] Image of an extended occlusion mask may be obtained as follows.

[0046] Position, size, shape and amount of light blocked by the occlusion mask may be defined based on the specified parameters. Size, shape and position of the occlusion mask may coincide with the size, shape and position of virtual image in the virtual image formation plane.

[0047] Withthe optical system magnification known, image of the occlusion mask may be formed in the image space 130. The resulting occlusion mask image in the image space 130 may be defined by the dependence of normalized distribution of light intensity on the point position in the image space 130.

[0048] The optical system PSF (Point Spread Function) may be determined for the specified distance from the optical system entrance pupil 150 to the electroactive panel 140. Using the obtained PSF, blur width of the occlusion mask image may be calculated. Operations for determining image blur width are known in the art and performed by computing systems e.g. a computer. Blur width may be defined by the dependence of normalized distribution of light intensity on the point position in the image space.

[0049] In an embodiment, image of extended occlusion mask may be obtained by adding the blur width defined by the dependence of normalized distribution of light intensity on the point position in the image space to the occlusion mask image defined by the dependence of normalized distribution of light intensity on the point position in the image space. It should be noted that for the same PSF, the blur width does not depend on the amount of light to be blocked.

[0050] Fig. 2 is a schematic view illustrating an extended occlusion mask image, and the dependence of normalized distribution of light intensity on the point position in the image space, defining this image.

[0051] Unblurred area 210 may correspond to virtual image 114, while all blur 220 is beyond the virtual image bounds. Beneath, Fig. 2 may show a plot of the dependence of normalized light intensity distribution on the point position in the image space, defining the extended occlusion mask image (shown by dotted line 240) and a plot of the dependence of normalized light intensity distribution on the point position in the image space, corresponding to the occlusion mask whose size, shape, and position coincide with the size, shape, and position of the virtual image in the virtual image formation plane (shown by solid line 250). 230 in the image space (Fig. 2) may correspond to the size and position of the extended occlusion mask image. Image blur is defined by the slope of dotted curve 242 in area 220. To eliminate the blur in the image space 130, the transmittance of elements in the extended occlusion mask area should be made such that the blur is reduced, that is, the area 220 in the image space 130 is reduced. The extended occlusion mask image may be determined by bound of the virtual object,

[0052] In an embodiment, the dependence of normalized distribution of light intensity on the point position in the extended occlusion mask image may be subjected to inverse convolution with PSF. The inverse convolution may result in the transmittance dependence on the point position in the extended occlusion mask on the electroactive panel 140, defining the extended occlusion mask area on the electroactive panel.

[0053] To reduce the blur effect of the occlusion mask image in the image space 130, it is proposed to divide the extended occlusion mask area on the electroactive panel into zones with different transmittance of elements, e.g. to perform zoning of the extended occlusion mask on the electroactive panel. The zoning may reduce the extended occlusion mask blur in the image space.

[0054] Fig. 3 (ring is shown as an example only, occlusion mask can have any shape corresponding to the virtual image shape) illustrates an embodiment of the location of zoned extended occlusion mask area on elements of the electroactive panel and the result of blur estimation. The virtual image bounds 330 coincide with the unblurred portion 320 of the zoned extended occlusion mask image.

[0055] Zone 1 of the zoned extended occlusion mask may be defined as the first zone 310, where elements are blocked and do not transmit light. In an embodiment, the amount of blocked light may be equal to predetermined amount of light to be blocked based on the real world brightness. The first zone 310 may be referred as a non-transparent zone and may be determined by the amount of blocked light.

[0056] Zone 2 of the zoned extended occlusion mask may be defined as the second zone 320, located inside the first zone 310. In an embodiment, elements in the second zone 320 may transmit light. The second zone 320 may be referred as a transparent zone, and may be determined by the size of the area needed to block light.

[0057] In an embodiment, the zoned extended occlusion mask may be defined by the transmittance dependence on the point position in the zoned extended occlusion mask on the electroactive panel. Next, the above dependence may be convoluted with PSF of the optical system used.

[0058] In an embodiment, the convolution may result in the dependence of normalized distribution of light intensity on the point position of the zoned extended occlusion mask in the image space 130, which defines the image of the zoned extended occlusion mask. In an embodiment, the dependence of normalized distribution of light intensity on the point position of the zoned extended occlusion mask in the image space is expressed as:

[0059] ... Equation (1)

[0060] where I(x,y) is the transmittance dependence on the point position in the zoned extended occlusion mask on the electroactive panel,

[0061] and arethe coordinates of points on the electroactive panel and the coordinates in the image space, respectively,

[0062] is PSF, denotes "Convolution", convolution operation.

[0063] The convolution operation may result in a zoned extended occlusion mask image, which will be seen by the eye, defined by .

[0064] In an embodiment, if the eye acts as an optical system, the convolution operation may result in a zoned extended occlusion mask image, which may be seen by the eye with a blur area, where the blur area 220 will be reduced, and this may be the remaining (unremoved) blur.

[0065] In an embodiment, error function E(I) may be determined between ideal occlusion mask (i.e. the dependence of normalized distribution of light intensity at each point of ideal occlusion mask on the point position in the image space, I'ideal), which has no blur in the image space, and the zoned extended occlusion mask in the image space (e.g. the dependence of normalized distribution of light intensity at each point of the resulting extended zoned occlusion mask on the point position in image space I').

[0066] In this case, error function will be:

[0067] (2)

[0068] where t (0, T), T is the number of electroactive panel elements.

[0069] The E(I) function may show the amountof difference between the calculated and ideal (required) result. Accordingly, the smaller the E(I), the closer the result is to the ideal.

[0070] In an embodiment, if the estimated error exceeds a predetermined threshold, the resulting zoned extended occlusion mask may be zoned again, but in a different way. Meanwhile, edges of Zone 1 310, where elements do not transmit light completely (blocked), may affect the residual blur of the zoned extended occlusion mask. Therefore, transmittance of the electroactive panel elements along the edge of Zone 1 310 may be changed element-by-element as in the steps above for each new zoned extended occlusion mask obtained after the element transmittance variation.

[0071] The steps described above may be repeated with each new zoned extended occlusion mask until the aforementioned error is less than a predetermined threshold. The zoned extended occlusion mask with the smallest error will be the resulting occlusion mask. The resulting occlusion mask is output on the electroactive panel for said virtual image.

[0072] If a virtual image is overlaid on the resulting zoned extended occlusion mask (hereinafter referred to as resulting occlusion mask), in some cases nonuniform distribution of light intensity may be visible, which may be eliminated. To do this, the central zone of the resulting occlusion mask may be changed.

[0073] In an embodiment, blur may be minimized by changing zone 1 element-by-element. In an embodiment, by changing element-by-element with attention to the bounds of the virtual image and blocked light level (e.g. the amount of blocked light), blur may be reduced. The residual blur of the zoned extended occlusion mask may be determined with transmittance variation which is calculated as above.

[0074] FIG. 4 illustrateshow image nonuniformity in the central part of the occlusion mask is reduced. Nonuniformity in the image in the central part of the occlusion mask can be reduced after obtaining the resulting occlusion mask.

[0075] Referring to FIG. 4, at 401, the transmittance dependence on the point position in the resulting occlusion mask is convoluted with PSF. At 402, the result is the dependence of normalized distribution of light intensity on the point position in the resulting occlusion mask image, which defines the resulting occlusion mask image. Based on the obtained image, central area L may be determined, where image nonuniformity may be visible. 402 shows the convolution result as the image produced by the optical system in the image space and as the plot of dependence of normalized distribution of light intensity on the point position in the image plane. Peak in the central area L may determine distribution of light intensity in the nonuniformity and its position. The data may be used for zoning the image nonuniformity area in the central area L of the extended occlusion mask. Next, the convolution may be performed again and the extent of nonuniformity reduction may be checked based on the error function between the ideal occlusion mask, having no image nonuniformity in the central area of the image, and the resulting occlusion mask after the latest zoning of the area with nonuniformity. Thus, at 404, the exhaustive search for blocking the elements on the electroactive panel may be used to reduce the error below the specified threshold.

[0076] In an embodiment, the central area of the resulting occlusion mask may be made non-transparent or semi-transparent, followed by the steps similar to those which reduce blur of the occlusion mask image, but with transmittance variation in the central zone of the occlusion mask. The zone transmittance may depend on the size of the virtual object and the amount of blocked light.

[0077] The number and order of zones in the occlusion mask, as well as the amount of light transmitted by the resulting occlusion mask may be any and may depend on the size and shape of the virtual image, and the amount of light to be blocked, which is known a priori. In an embodiment, at any zoning of the occlusion mask for the same virtual image and the same brightness, the amount of blocked light may be known a priori and may not change. The number and order of zones in the occlusion mask may depend on the size of the virtual object and the amount of blocked light.

[0078] FIG. 5 shows a schematic structure diagram of an electronic device to which an embodiment of the present disclosure is applied. As shown in FIG. 5, the electronic device 5000 shown in FIG. 5 may include a processor 5001 and a memory 5003. The processor 5001 is connected to the memory 5003, for example, through a bus 5002. In an example, the electronic device 5000 may further include a transceiver 5004, and the transceiver 5004 may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that, in practical applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation to the embodiments of the present disclosure. In an example, the electronic device may be a first network node, a second network node or a third network node.

[0079] The processor 5001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure. The processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0080] The bus 5002 may include a path to transfer information between the components described above. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus or the like. The bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in FIG. 5, but it does not mean that there is only one bus or one type of bus.

[0081] The memory 5003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, and can also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, compact disk storage (including compressed compact disc, laser disc, compact disc, digital versatile disc, blue-ray disc, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation.

[0082] The memory 5003 is used for storing computer programs for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 5001 is configured to execute the computer programs stored in the memory 5003 to implement the operations shown in the foregoing method embodiments.

[0083] Embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, the computer program, when executed by a processor, implements the operations and corresponding contents of the foregoing method embodiments.

[0084] Embodiments of the present disclosure also provide a computer program product including a computer program, the computer program when executed by a processor realizing the operations and corresponding contents of the preceding method embodiments.

[0085] FIG. 6A and FIG. 6B illustrates a usability and use cases of the present disclosure. FIG. 6A illustrates, in an embodiment, the image before using the present disclosure 610 and the image after using the present disclosure 620. In an embodiment, the proposed solution may make it possible for a user to use AR glasses in bright light conditions and see a colorful virtual image. In an embodiment, the proposed solution may make it possible for a user to see virtual image without optical occlusion blur. In an embodiment, the proposed solution may make it possible for a user to feel comfortable with portable head-mounted device. FIG 6B illustrates, in an embodiment, use cases of the present disclosure. The present disclosure may be used in augmented reality glasses with any brightly illuminated images of real world, all of real, virtual and mixed (AR glasses 632 / VR glasses 652 / MR glasses 642), but not limited thereto. As shown in 620, The disclosure may reduce or eliminate blur of an occlusion mask of any shape and size. The electronic device for occlusion mask blur reduction can be used with any augmented reality glasses and helmets, anywhere, including outdoors in sunny weather; owing to eliminated blur, the virtual image will not be distorted by various bright objects.

[0086] Although the invention has been described in connection with some illustrative embodiments, it should be understood that the essence of the invention is not limited to these specific embodiments. On the contrary, it is assumed that the invention includes all alternatives, amendments and equivalents that may be included within the essence and scope of the claims.

[0087] Furthermore, the invention retains all equivalents of the claimed invention even if the claims are amended during prosecution.

[0088] According to an embodiment of the disclosure, the electronic device may be augmented reality (AR) glasses.

[0089] According to an embodiment of the disclosure, the method may include determining blur width of the occlusion mask image. The method may include obtaining an extended occlusion mask image by adding the determined blur width to the occlusion mask image.

[0090] According to an embodiment of the disclosure, the determining blur width of the occlusion mask image may include determining PSF (point spread function) of the optical system for a distance from the optical system entrance pupil to the electroactive panel. The determining blur width of the occlusion mask image may include determining the blur width of the occlusion mask image, based on the PSF of the optical system.

[0091] According to an embodiment of the disclosure, the obtaining the extended occlusion mask may include performing, based on the PSF, inverse convolution of the dependence of normalized distribution of light intensity of point position in the extended occlusion mask. The obtaining the extended occlusion mask may include obtaining transmittance dependence on the point position in the extended occlusion mask on the electroactive panel. The obtaining the extended occlusion mask may include determining position of the extended occlusion mask on the electroactive panel.

[0092] According to an embodiment of the disclosure, the method may include determining a plurality of zones of the extended occlusion mask to obtain a zoned extended occlusion mask. The plurality of zones may be with difference transmittance of elements on the electroactive panel.

[0093] According to an embodiment of the disclosure, the method may include convoluting the transmittance dependence on the point position in the zoned extended occlusion mask with the point spread function (PSF) of the optical system. The method may include determining blur area of the zoned extended occlusion mask image.

[0094] According to an embodiment of the disclosure, the method may include determining the error function between an ideal occlusion mask and the zoned extended occlusion mask. The method may include changing element-by-element the transmittance of electroactive panel elements until the error is less than a predetermined threshold. The method may include determining the resulting occlusion mask on the electroactive panel for the virtual image. The ideal occlusion mask may have no blur in the image space.

[0095] According to an embodiment of the disclosure, the determining the plurality of zones of the extended occlusion mask may include dividing the electroactive panel area corresponding to the virtual image into a plurality of zones such that the total light transmittance of elements of the zones is equal to the amount of light blocked by the occlusion mask.

[0096] The method may include eliminating nonuniformity of the central area of the resulting occlusion mask.

[0097] According to an embodiment of the disclosure, the eliminating nonuniformity of the central area of the resulting occlusion mask may include convoluting the transmittance dependenceon the point position in the resulting occlusion mask with the PSF to obtain the dependence of normalized distribution of light intensity on the point position in the resulting occlusion mask image. The method may include determining, from the resulting occlusion mask image, the position of nonuniformity of the central area, and the corresponding position of the nonuniformity area on the electroactive panel.

[0098] According to an embodiment of the disclosure, the eliminating nonuniformity of the central area of the resulting occlusion mask may include determining a plurality of zones of the nonuniformity area on the electroactive panel. The eliminating nonuniformity of the central area of the resulting occlusion mask may include determining the error function between the ideal occlusion mask, and the zoned resulting occlusion mask. The plurality of zones are with difference transmittance of elements on the electronic panel. The ideal occlusion mask may have no image nonuniformity in the image central area;

[0099] According to an embodiment of the disclosure, the method may include identifying the error is less than a predetermined threshold. The method may include convoluting the transmittance dependence on the point position in the resulting occlusion mask with the PSF based on the determination that the error is not less than the predetermined threshold. The method may include obtaining the resulting occlusion mask without central area nonuniformity on the electroactive panel for the virtual image.

[0100] According to an embodiment of the disclosure, a device for blur reduction of an occlusion mask used for a virtual image displayed against real world image with augmented reality (AR) glasses is disclosed herein. The device may include, arranged along an optical axis, an optical system for producing a visible image in image space. The device may include, an entrance pupil of the optical system. The device may include AR glasses with a screen parallel to the entrance pupil plane, wherein a virtual image located on the AR glasses screen has a specified position on the screen, a specified shape, and a specified size. The device may include an electroactive panel parallel to the AR glasses screen, the electroactive panel being placed at a known distance from the plane of the optical system entrance pupil and at a known distance from the plane of the AR glasses screen, where the virtual image is located. The device may include real world. The electroactive panel may be a screen consisting of elements. Each electroactive panel element may be either open, where the element transmits surrounding light, or closed, where the element does not transmit surrounding light, or can partially transmit surrounding light. The light transmittance of each element may be controlled by signals to each element. The electroactive panel may be configured to form a resulting occlusion mask for forming an occlusion area in the virtual image space by changing the light transmittance of said elements. Bounds of the occlusion area may coincide with bounds of the virtual image. The transmittance of the electroactive panel elements of the resulting occlusion mask in the occlusion area may be such that the error function between an ideal occlusion mask, which has no blur in the image space, and said resulting occlusion mask is less than a predetermined threshold.

[0101] The optical system may be user's eye. The optical system may be a camera.

[0102] According to an embodiment of the disclosure, a method for operating a device for blur reduction of an occlusion mask used for a virtual image displayed against real world image with augmented reality (AR) glasses is disclosed herein. The method may include determining position, size, shape and amount of light blocked by the occlusion mask. The position, size and shape of the occlusion mask may depend on position, size and shape of the predetermined virtual image. The amount of light blocked by the occlusion mask may depend on surrounding brightness. The method may include producing, by the optical system, an occlusion mask image in the image space. The method may include determining PSF (Point Spread Function) of the optical system for a specified distance from the optical system entrance pupil to the electroactive panel. The method may include estimating, based on the PSF, blur width of the occlusion mask image. The method may include adding the resulting blur width to the occlusion mask image to obtain an extended occlusion mask image.The method may include performinginverse convolution of the dependence of normalized distribution of light intensity on the point position in the extended occlusion mask image with the PSF to obtain the transmittance dependence on the point position in the extended occlusion mask on the electroactive panel, defining the extended occlusion mask area on the electroactive panel. The method may include zoning the extended occlusion mask area on the electroactive panel by creating at least two zones with different transmittance of elements on the electroactive panel to obtain a zoned extended occlusion mask. The method may include determining blur area of the zoned extended occlusion mask image produced by the optical system in the image space by convoluting the transmittance dependence on the point position in the zoned extended occlusion mask with the point spread function (PSF) of the optical system. The method may include determining the error function between an ideal occlusion mask, that has no blur in the image space, and the zoned extended occlusion mask. The method may include changing element-by-element the transmittance of electroactive panel elements until said error is less than a predetermined threshold to obtain thereby a resulting occlusion mask. The method may include forming the resulting occlusion mask on the electroactive panel for said virtual image.

[0103] According to an embodiment of the disclosure, the zoning the extended occlusion mask area may include dividing the electroactive panel area corresponding to the virtual image into at least two zones such that the total light transmittance of elements of said zones is equal to the amount of light blocked by the occlusion mask.

[0104] According to an embodiment of the disclosure, the method may include eliminating nonuniformity of the central area of the resulting occlusion mask. The eliminating nonuniformity of the central area of the resulting occlusion mask may include convoluting the transmittance dependenceon the point position in the resulting occlusion mask with the PSF to obtain the dependence of normalized distribution of light intensity on the point position in the resulting occlusion mask image, defining the resulting occlusion mask image. The eliminating nonuniformity of the central area of the resulting occlusion mask may include determining, from the resulting occlusion mask image, the position of nonuniformity area in the image central area, and the corresponding position of said nonuniformity area on the electroactive panel. The eliminating nonuniformity of the central area of the resulting occlusion mask may include zoning, on the electroactive panel, said nonuniformity area by creating at least two zones having different transmittance of elements on the electroactive panel. The eliminating nonuniformity of the central area of the resulting occlusion mask may include determining the error function between an ideal occlusion mask, that has no image nonuniformity in the image central area, and the zoned resulting occlusion mask. The eliminating nonuniformity of the central area of the resulting occlusion mask may include repeating above steps with the zoned resulting occlusion mask as the resulting occlusion mask until said error is less than a predetermined threshold, to obtain thereby a resulting occlusion mask without central area nonuniformity. The eliminating nonuniformity of the central area of the resulting occlusion mask may include forming the resulting occlusion mask without central area nonuniformity on the electroactive panel for said virtual image.

Claims

1.An electronic device for blur reduction of an occlusion mask, the electronic device comprising:an optical system;memory configured to store instructions; andone or more processors;wherein the instructions, when executed by the one or more processors, cause the electronic device to:determine position, size, shape and amount of light blocked by the occlusion mask, wherein the position, size and shape of the occlusion mask is determined based on position, size and shape of a virtual image, and wherein the amount of light blocked by the occlusion mask is determined based on surrounding brightness;obtain, by the optical system, an occlusion mask image in an image space; andchange transmittance of electroactive panel to reduce remaining blur by transmittance variation.2.The electronic device according to claim 1, wherein the electronic device is augmented reality (AR) glasses.3.A method for blur reduction of an occlusion mask comprising:determining position, size, shape and amount of light blocked by the occlusion mask, wherein the position, size and shape of the occlusion mask is determined based on position, size and shape of a virtual image, and the amount of light blocked by the occlusion mask is determined based on surrounding brightness;obtaining, by an optical system, an occlusion mask image in an image space; andchanging transmittance of electroactive panel to reduce remaining blur by transmittance variation.4.The method according to claim 3, further comprising:determining blur width of the occlusion mask image; andobtaining an extended occlusion mask image by adding the determined blur width to the occlusion mask image.5.The method according to any one of claims 3 to 4, wherein the determining blur width of the occlusion mask image comprises:determining PSF (point spread function) of the optical system for a distance from the optical system entrance pupil to the electroactive panel; anddetermining the blur width of the occlusion mask image, based on the PSF of the optical system.6.The method according to any one of claims 3 to 5, wherein the obtaining the extended occlusion mask comprises:performing, based on the PSF, inverse convolution of the dependence of normalized distribution of light intensity of point position in the extended occlusion mask;obtaining transmittance dependence on the point position in the extended occlusion mask on the electroactive panel; anddetermining position of the extended occlusion mask on the electroactive panel.7.The method according to any one of claims 3 to 6, further comprising:determining a plurality of zones of the extended occlusion mask to obtain a zoned extended occlusion mask;wherein the plurality of zones are with difference transmittance of elements on the electroactive panel.8.The method according to any one of claims 3 to 7, further comprising:convoluting the transmittance dependence on the point position in the zoned extended occlusion mask with the point spread function (PSF) of the optical system; anddetermining blur area of the zoned extended occlusion mask image.9.The method according to any one of claims 3 to 8, further comprising:determining the error function between an ideal occlusion mask and the zoned extended occlusion mask;changing element-by-element the transmittance of electroactive panel elements until the error is less than a predetermined threshold; anddetermining the resulting occlusion mask on the electroactive panel for the virtual image.wherein the ideal occlusion mask has no blur in the image space.10.The method according to any one of claims 3 to 9, wherein the determining the plurality of zones of the extended occlusion mask comprises:dividing the electroactive panel area corresponding to the virtual image into a plurality of zones such that the total light transmittance of elements of the zones is equal to the amount of light blocked by the occlusion mask.11.The method according to any one of claims 3 to 10, further comprising:eliminating nonuniformity of the central area of the resulting occlusion mask.12.The method according to any one of claims 3 to 11, wherein the eliminating nonuniformity of the central area of the resulting occlusion mask comprises:convoluting the transmittance dependenceon the point position in the resulting occlusion mask with the PSF to obtain the dependence of normalized distribution of light intensity on the point position in the resulting occlusion mask image; anddetermining, from the resulting occlusion mask image, the position of nonuniformity of the central area, and the corresponding position of the nonuniformity area on the electroactive panel.13.The method according to any one of claims 3 to 12, wherein the eliminating nonuniformity of the central area of the resulting occlusion mask comprises:determining a plurality of zones of the nonuniformity area on the electroactive panel; anddetermining the error function between the ideal occlusion mask, and the zoned resulting occlusion mask;wherein the plurality of zones are with difference transmittance of elements on the electronic panel; andwherein the ideal occlusion mask has no image nonuniformity in the image central area.14.The method according to any one of claims 3 to 13, further comprising:identifying the error is less than a predetermined threshold;convoluting the transmittance dependence on the point position in the resulting occlusion mask with the PSF based on the determination that the error is not less than the predetermined threshold; andobtaining the resulting occlusion mask without central area nonuniformity on the electroactive panel for the virtual image.15.A computer-readable medium configured to store instructions which, when executed by at least one processor of an electronic device, cause the electronic device to perform the method of any one of claims 3 to 14.

Images