MR device and method for eliminating image flicker of MR device
By adjusting the frequency of the camera module and the rendering module in the MR device, matching the ambient light flicker frequency, and generating intermediate images to adapt to the operating frequency of the display module, the image flicker problem caused by ambient light flicker in the prior art is solved, and the mixed reality is improved Refresh rate and user experience of the device.
Patent Information
- Application Number
- CN202211021642.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-24
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-08-24
AI Technical Summary
Existing MR products cannot properly set display, camera and rendering parameters under ambient light flicker frequency conditions, resulting in image flickering problems not being effectively solved. Adjusting the camera module parameters will cause the camera module and the display module to be incompatible with the operating frequency, resulting in display Image misalignment, frame skipping and other problems.
By adjusting the shooting parameters of the camera module in the MR device, matching the ambient light flickering frequency, generating intermediate images to adapt to the operating frequency of the display module, ensuring that the target shooting frequency of the camera module is not greater than the flash frequency, and controlling the display module at a specific operating frequency Display the target image and use the rendering module to match the rendering frequency to solve the image flickering problem caused by frequency mixing.
Effectively solve the problem of image flicker caused by ambient light flicker, avoid misalignment and frame skipping caused by mismatch between the camera module and the display module, improve the refresh rate and user experience of mixed reality equipment, dynamically adapt to ambient light changes, and improve the Mixed reality performance.
Smart Images

Figure CN117676114B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to an MR device and a method for eliminating image flicker of the MR device. Background Art
[0002] The settings of GPU rendering refresh rate, display refresh rate, camera sensor refresh rate and exposure time of the existing mixed reality (MR) products on the market are formulated based on multiple user experience indicators and software and hardware performance limitations, but the influence of the frequency of ambient light changes and the frequency of brightness changes of the photographed objects are not taken into account. Therefore, under different ambient light flicker frequency conditions, the display, camera, and rendering parameters of existing MR products will not be properly set to avoid flicker visible to the naked eye. This is because the existing products on the market generally do not provide high-performance mixed reality or perspective functions, and the display and camera refresh rates generally do not exceed the multiple of the mains frequency. Therefore, the problem of image flicker caused by ambient light flicker has not received enough attention. In the next generation of high-performance mixed reality products, as the requirements for display images and user experience gradually increase, this problem must be properly solved. Summary of the invention
[0003] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.
[0004] In order to solve the problem of image flicker caused by ambient light flicker, the relevant shooting parameters of the camera module can be adjusted to match the ambient light flicker, thereby avoiding the image flicker problem caused by ambient light flicker. However, since the MR device includes not only a camera module, but also a display module, etc. Although changing the relevant shooting parameters of the camera module can overcome the problem of screen flicker when generating the original image, it causes the problem of mismatch in working frequency between the camera module and the display module, which will eventually still be reflected in the target image displayed by the display module. There will still be misalignment, frame skipping, etc., and the problem of flickering in the image finally seen by the user through the display device of the MR device is still not improved.
[0005] In order to solve the image flicker problem caused by not considering ambient light flicker in VR, AR or MR products in the prior art, and at the same time solve the secondary problem of display image flicker or instability caused by camera, rendering, and display frequency mixing in order to solve the ambient light flicker problem, the present invention provides an MR device and a method for eliminating image flicker of the MR device.
[0006] An MR device provided by one aspect of the present invention comprises at least a camera module and a display module. Another aspect of the present invention provides a method for eliminating image flicker of an MR device, comprising: in response to the presence of ambient light flicker that triggers human eye perception, determining the flicker frequency of the ambient light flicker; judging whether the display module has a specific operating frequency according to the operating state of the MR device; in response to the presence of only a single flicker frequency and the presence of the specific operating frequency greater than the flicker frequency, always maintaining the target shooting frequency of the camera module not greater than the flicker frequency; based on the original image acquired by the camera module at the target shooting frequency, generating an intermediate image corresponding to the specific operating frequency; and controlling the display module to display the target image obtained based on the intermediate image at the specific operating frequency.
[0007] In one embodiment of the above method, optionally, generating an intermediate image corresponding to the specific operating frequency based on the original image acquired by the camera module at the target shooting frequency further includes: at the target moment corresponding to the specific operating frequency, generating the intermediate image of the target moment based on two frames of original images before and after the target moment.
[0008] In one embodiment of the above method, optionally, generating an intermediate image corresponding to the specific operating frequency based on the original image acquired by the camera module at the target shooting frequency further includes: at the target moment corresponding to the specific operating frequency, based on the change law of the original image of a predetermined number of frames before the target moment, predicting the intermediate image at the target moment based on the previous frame of the original image at the target time.
[0009] In one embodiment of the above method, optionally, generating an intermediate image corresponding to the specific operating frequency based on the original image acquired by the camera module at the target shooting frequency further includes: acquiring the user's field of view movement data according to the positioning sensor of the MR device; and at a target moment corresponding to the specific operating frequency, performing at least one of translation, scaling, and distortion processing on the previous frame of the original image at the target time based on the field of view movement data to generate an intermediate image at the target moment.
[0010] In an embodiment of the above method, optionally, always keeping the target shooting frequency of the camera module not greater than the flickering frequency further includes: always keeping the target shooting frequency of the camera module at the flickering frequency.
[0011] In one embodiment of the above method, optionally, always keeping the target shooting frequency of the camera module not greater than the flicker frequency further includes: always keeping the target shooting frequency of the camera module less than the flicker frequency, and adjusting the target shooting frequency of the camera module so that the specific operating frequency is a multiple of the target shooting frequency.
[0012] In an embodiment of the above method, optionally, in response to the target shooting frequency being less than the flickering frequency, the exposure time of the camera module is adjusted to match the flickering period of the ambient light flicker.
[0013] In an embodiment of the above method, optionally, the MR device further includes a rendering module, and the method further includes: determining whether the rendering module has a specific rendering frequency; wherein in response to the existence of a specific rendering frequency in the rendering module, and the specific rendering frequency does not match the specific operating frequency, controlling the rendering module to generate a rendering image with the specific rendering frequency, generating an intermediate rendering image corresponding to the specific operating frequency based on the rendering image with the specific rendering frequency, and the target image is also obtained based on the intermediate rendering image; or in response to the absence of a specific rendering frequency in the rendering module, controlling the rendering module to generate a rendering image with the specific operating frequency, and the target image is also obtained based on the intermediate rendering image.
[0014] In an embodiment of the above method, optionally, judging whether the display module has a specific operating frequency according to the working state of the MR device further includes: judging whether the display module has a specific operating frequency according to the working mode of the MR device and the display content of the display module; wherein in response to the MR device being in an energy-saving mode, the display module has a specific operating frequency less than a first preset threshold; in response to the MR device being in a high-performance mode, the display module has a specific operating frequency greater than a second preset threshold; in response to the display content of the display module including a video source of a specific frequency, the flickering frequency of the display module is a multiple of the specific frequency of the video source.
[0015] Another aspect of the present invention provides an MR device, which includes at least a camera module and a display module, wherein the MR device also includes at least one processor; and a memory coupled to the at least one processor, the memory containing instructions stored therein, and when the instructions are executed by the at least one processor, the MR device executes the method for eliminating image flicker of the MR device as described in any one of the embodiments above.
[0016] Another aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for eliminating image flicker of an MR device as described in any one of the above embodiments is implemented.
[0017] According to the MR device and the method for eliminating image flicker of the MR device provided by the present invention, the problem of screen flicker or instability caused by changes in ambient light brightness and mixing of camera, rendering, and display frequency can be solved by adjusting the shooting parameters of the camera module and generating an intermediate image based on the original image of the camera module. The present invention provides a solution for camera and display devices to adopt any refresh rate, which can dynamically adapt to changes in ambient light on the one hand, and on the other hand, provides the possibility of increasing the refresh rate of the display device, that is, it lays a foundation for further improving the performance of mixed reality. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.
[0019] Figure 1 A schematic structural diagram of an MR device provided in one aspect of the present invention is shown.
[0020] Figure 2 A schematic flow chart of a method for eliminating image flicker in an MR device provided by another aspect of the present invention is shown.
[0021] Figure 3 The figure shows the relationship between the ambient light flickering period, the display module frequency, and the camera module frequency in the present invention. DETAILED DESCRIPTION
[0022] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.
[0023] The following description is given to enable those skilled in the art to implement and use the invention and incorporate it into a specific application context. Various modifications, as well as various uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein are applicable to a wide range of embodiments. Thus, the present invention is not limited to the embodiments given herein, but should be granted the broadest scope consistent with the principles and novel features disclosed herein.
[0024] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the practice of the present invention need not be limited to these specific details. In other words, known structures and devices are shown in block diagram form without detailed display to avoid obscuring the present invention.
[0025] The reader is reminded of all documents and literature filed simultaneously with this specification and open to public inspection, and the contents of all such documents and literature are incorporated herein by reference. Unless otherwise directly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features for achieving the same, equivalent or similar purposes. Therefore, unless otherwise explicitly stated, each feature disclosed is only an example of a group of equivalent or similar features.
[0026] Note that, where used, the symbols left, right, front, back, top, bottom, forward, reverse, clockwise and counterclockwise are used only for convenience and do not imply any specific fixed direction. In fact, they are used to reflect the relative position and / or direction between the various parts of the object. In addition, the terms "first" and "second" are used only for descriptive purposes and should not be understood as indicating or implying relative importance.
[0027] Note that, in the case of use, further, preferably, further and more preferably are simple beginnings for explaining another embodiment based on the previous embodiment, and the content of further, preferably, further or more preferably followed by the above embodiment is combined with the previous embodiment as a complete composition of another embodiment. Several further, preferably, further or more preferably settings followed by the same embodiment can be arbitrarily combined to form another embodiment.
[0028] In order to solve the image flicker problem caused by ambient light flicker in MR products, the present invention provides an MR device and a method for eliminating image flicker in the MR device. Figure 1 To understand the MR device provided by one aspect of the present invention, and in combination with Figure 1 To understand the application scenario of the method for eliminating image flicker of an MR device provided by another aspect of the present invention.
[0029] like Figure 1 As shown, the MR device provided in one aspect of the present invention at least includes a camera module 100 and a display module 400. The MR device processes the images captured by the camera module 100 and displays them in the display module 400.
[0030] In such Figure 1In the illustrated embodiment, the camera module 100 includes a camera sensor 110 and an ISP image processing module 120. The exposure time and shooting frequency of the camera sensor 110 are adjustable, and the processing frequency of the ISP image processing module 120 matches the shooting frequency of the camera sensor 110. The above-mentioned specific composition of the camera module 100 is only illustrative and should not unduly limit the scope of protection of the present invention.
[0031] In such Figure 1 In the illustrated embodiment, the display module 400 includes a display image processing module 410 and a display screen 420. The frequency at which the display image processing module 410 processes the target image to be displayed is adjustable, and the refresh rate of the display screen 420 matches the processing frame rate of the display image processing module 410.
[0032] In one embodiment, the MR device processes the original image captured by the camera module 100 through the rendering module 500 and the image fusion module 300. That is, the image fusion module 300 fuses the digitized real image output by the camera module 100 and the virtual digital image output by the rendering module 500 to enhance and render the image acquired by the camera module 100, and outputs the mixed target image for display.
[0033] Ambient light can be affected by electronic light-emitting devices, such as artificial lighting and electronic display devices. Electronic devices generally have periodic modulation of brightness. For example, when the AC frequency of the mains is 50Hz, LED lights and fluorescent lights have a high-frequency brightness flicker of 100Hz. Although the human eye cannot see flickers above 90Hz, when the refresh rate of the camera device mixes with the ambient light flicker frequency, new low-frequency components will be produced. For example, when shooting an environment with 100Hz LED lighting at a camera frequency of 120Hz, a 20Hz difference frequency flicker will be seen. Under existing products and designs, if the display refresh rate of the MR device is higher than 100Hz, or the camera exposure time is set to less than 100ms, difference frequency flicker will occur without compensation function.
[0034] Please combine Figure 2 To further understand the method for eliminating MR device image flicker provided by one aspect of the present invention. Figure 2 As shown, one aspect of the present invention provides a method comprising:
[0035] Step S100: determine whether there is ambient light flickering that triggers human eye perception, and in response to the presence of flickering, execute step S200: determine the flickering frequency of the ambient light flickering.
[0036] In step S300 , it is determined whether the display module has a specific operating frequency, and in step S400 , it is determined whether the flickering frequency is single and whether the specific operating frequency is greater than the flickering frequency.
[0037] In response to the flickering frequency being single and the specific operating frequency being greater than the flickering frequency, step S500 is executed to always keep the target shooting frequency of the camera module no greater than the flickering frequency; step S600 is to generate an intermediate image corresponding to the specific operating frequency based on the original image acquired by the camera module at the target shooting frequency; and step S700 is to control the display module to display the target image obtained based on the intermediate image at the specific operating frequency.
[0038] In step S100, the ambient light flicker that triggers human eye perception refers to light flicker below 80-90 Hz. Although the human eye is a very complex system and has great individual differences, for most people, light flicker below 80-90 Hz can be perceived. At this time, the human eye can perceive changes in light and dark. Flicker frequencies below 80-90 Hz are also called visible flicker. Visible flicker is unacceptable for ordinary lighting or display, and is also expected to be eliminated in MR devices.
[0039] According to one embodiment, in step S100, the ambient light sensing sensor or the camera module 100 may sense and analyze the periodic changes of the full field of view or a part of the ambient light. At the same time, when judging whether the periodic changes of the ambient light will trigger the perception of the human eye, it is determined whether the full field of view or a part of the ambient light has periodic changes within a specific frequency range and the amplitude of the changes exceeds a specific threshold value, so as to comprehensively determine whether the ambient light will trigger the perception of the human eye.
[0040] In the present invention, in response to the presence of ambient light flickering that triggers human eye perception, it is first necessary to determine the flickering frequency of the ambient light flickering. Those skilled in the art may use existing or future technical means to determine the flickering frequency of the ambient light flickering, and the specific determination of the flickering frequency of the ambient light flickering should not unduly limit the protection scope of the present invention.
[0041] Furthermore, when there is ambient light flickering that triggers human eye perception, it is necessary to further determine whether the flickering frequency of the ambient light flickering that triggers human eye perception is single and whether it is less than the operating frequency of the display module of the MR device.
[0042] The single flicker frequency of the ambient light flicker means that in step S200, only a unique flicker frequency of the ambient light flicker is determined, rather than multiple flicker frequencies. The operating frequency of the display module includes the refresh rate of the display device (the number of frames displayed by the display device per second, or the reciprocal 1 / T of the time T that each frame stays on the screen), and the display signal processing frame rate (the number of frames processed by the display image processing circuit per second, or the reciprocal 1 / T of the time T required for each frame image processing). Generally speaking, the refresh rate of the display device and the display signal processing frame rate remain the same.
[0043] In one embodiment, first, in step S300, it is necessary to determine whether the display module has a specific operating frequency according to the working state of the MR device. Further, in one embodiment, it is determined whether the display module has a specific operating frequency according to the working mode of the MR device and the display content of the display module. In response to the MR device being in energy-saving mode, the display module has a specific operating frequency less than a first preset threshold. In response to the MR device being in high-performance mode, the display module has a specific operating frequency greater than a second preset threshold. In response to the display content of the display module including a video source of a specific frequency, the specific operating frequency of the display module is a multiple of the specific frequency of the video source.
[0044] As described above, when a user uses the MR device, the most intuitive feeling is the visual experience, so the working state of the display module will determine the user experience. To this end, if the display module already has a specific working mode, even if there is ambient light flickering that will cause screen flickering, it is still necessary to maintain the display module in the specific working mode, that is, in step S700, always control the display module to display the target image at a specific working frequency.
[0045] Since it is determined in step S400 that the ambient light flicker frequency is single and less than the specific operating frequency of the display module, it means that the ambient light flicker frequency is not equal to the specific operating frequency and there is a frequency difference between the two. If the shooting frequency of the camera module is related to the specific operating frequency, the problem of screen flickering will be caused due to the frequency difference between the shooting frequency and the ambient light flicker frequency.
[0046] According to one aspect of the present invention, by executing step S500, the target shooting frequency of the camera module is made not greater than the flicker frequency, that is, the working parameters of the camera module dynamically follow the changes in the ambient light flicker frequency, thereby solving the problem of screen flickering in the original image obtained by the camera module from the source.
[0047] However, this gives rise to another problem, that is, since the shooting frequency of the original image generated by the camera module 100 and the display frequency of the image displayed by the display module 400 cannot match each other, based on the existing MR device, the picture finally displayed by the display module 400 will cause problems such as picture misalignment, frame skipping, and incoherence due to the mismatch between the shooting frequency and the display frequency, and will also cause the user to visually feel that the display picture is "flickering".
[0048] To this end, the image adaptation module 210 needs to execute step S600 to generate an intermediate image corresponding to a specific working frequency based on the original image acquired by the camera module at the target shooting frequency, and the target image displayed by the display module is acquired based on the intermediate image.
[0049] Please combine Figure 3 To understand the relationship between the ambient light flicker cycle and the operating frequency of the display module and the camera module, and thus to understand how to generate an intermediate image corresponding to a specific operating frequency based on the original image acquired by the camera module at the target shooting frequency in step S600. Figure 3 In step S600, it is necessary to obtain intermediate images at moments B1, B2, B3, B4, B5, ... corresponding to the operating frequency of the display module based on the original images generated by the camera module at moments A1, A2, A3, A4, ...
[0050] In one embodiment, step S600 further includes: at a target moment corresponding to a specific operating frequency, generating an intermediate image at the target moment based on two frames of original images before and after the target moment.
[0051] In the above embodiment, an intermediate image corresponding to a specific operating frequency is generated based on the original image acquired by the camera module at the target shooting frequency by inter-frame interpolation. In this embodiment, the intermediate image at time B2 needs to be generated based on the original images at time A1 and A2.
[0052] In this embodiment, since the original images before and after the B2 moment are determined, the intermediate image at the B2 moment can be determined by the change between the two frames of images A1 and A2. And since the two frames of images A1 and A2 are determined, the accuracy of the intermediate image at the B2 moment determined according to this embodiment is relatively high.
[0053] It is understandable that, since the confirmation of the intermediate image at the moment B2 requires the original images at the moments A1 and A2, and the moment A2 is later than the moment B2, based on this embodiment, the target image displayed by the display module is slightly delayed from the original image acquired by the camera module as a whole. It is understandable that, even in the prior art, since the camera module needs to output the image to the display module, and there may be other image processing processes in this process, the display module is inevitably slightly delayed from the original image acquired by the camera module when displaying the target image, and the delay for generating B2 based on A1 and A2 does not affect the overall delay situation, and can still be controlled within a reasonable delay range.
[0054] In another embodiment, step S600 further includes: at a target moment corresponding to a specific operating frequency, based on a change rule of a predetermined number of frames of original images before the target moment, predicting an intermediate image at the target moment based on a previous frame of original image at the target time.
[0055] For example, in this embodiment, in order to obtain the intermediate image at time B4, it is first necessary to collect the change law of images A1, A2, and A3 (the predetermined number of frames in this example is 3, which is only for illustration and should not unduly limit the scope of protection of the present invention), and according to the above change law, based on the original image at time A3, predict the intermediate image at time B4.
[0056] In this embodiment, the intermediate image of the target time is obtained based on the historical original image, and there is no need to wait until a frame of image is generated after the target time, so the delay is small. Moreover, based on the change law of the original image of the predetermined number of frames, a relatively accurate intermediate image can also be obtained when the scene does not suddenly change.
[0057] In another embodiment, step S600 further includes: acquiring the user's field of view movement data according to the positioning sensor of the MR device; and at a target moment corresponding to a specific operating frequency, performing at least one of translation, scaling, and distortion processing on the previous frame of the original image at the target time based on the field of view movement data to generate an intermediate image at the target moment.
[0058] For example, in this embodiment, in order to obtain the intermediate image at time B3, the original image at time A2 is first required, and the user's field of view movement data is also required to be obtained through the positioning sensor of the MR device. The positioning sensor may include an eye movement sensor and / or a head positioning sensor. The eye movement sensor can provide the movement data of the user's gaze point, and the head positioning sensor can output the three-degree-of-freedom information of the user's head, namely X / Y / Z and the corresponding calculated yaw, pitch and roll information, and can also position and output the six-degree-of-freedom information of the user's head, including (1) forward / backward, (2) up / down, (3) left / right, (4) yaw, (5) pitch and (6) roll, thereby providing the movement data of the user's head.
[0059] The user's field of view movement data can be comprehensively acquired through the gaze point movement data of the eye movement sensor and / or the movement data of the user's head provided by the head positioning sensor, so that an intermediate image at moment B3 can be generated based on the original image at moment A2 through at least one of translation, scaling, and distortion processing.
[0060] In this embodiment, the intermediate image of the target time is obtained based on the historical original image, and there is no need to wait until a frame of image is generated after the target time, so the delay is small. At the same time, since the user's field of view movement data is taken into account, a relatively accurate intermediate image can also be obtained when the scene does not change suddenly, and the intermediate image is an intermediate image that reflects the user's movement.
[0061] Through step S600, the problem of frame skipping and discontinuity of displayed images caused by adjusting the working parameters of the camera module to flicker with the ambient light and failing to match the display module can be solved. Therefore, the present invention provides a solution for the camera and display devices to adopt any refresh rate (the two may not match), which can dynamically adapt to changes in ambient light on the one hand, and further improve the refresh rate of the display device on the other hand, thus laying a foundation for further improving the performance of mixed reality.
[0062] In one embodiment, in step S500, always keeping the target shooting frequency of the camera module not greater than the flicker frequency further includes: always keeping the target shooting frequency of the camera module at the flicker frequency. By keeping the target shooting frequency consistent with the flicker frequency, the original image obtained can be free of screen flicker due to the lack of frequency difference.
[0063] In another embodiment, in step S500, always keeping the target shooting frequency of the camera module not greater than the flicker frequency further includes: always keeping the target shooting frequency of the camera module less than the flicker frequency, and adjusting the target shooting frequency of the camera module so that the flicker frequency is a multiple of the target shooting frequency. Since the flicker frequency is a multiple of the target shooting frequency, the original image obtained can also be free of screen flicker.
[0064] In another embodiment, in step S500, always keeping the target shooting frequency of the camera module not greater than the flicker frequency further includes: always keeping the target shooting frequency of the camera module less than the flicker frequency, and adjusting the target shooting frequency of the camera module so that the flicker frequency is a multiple of the target shooting frequency.
[0065] Furthermore, in this embodiment, it is necessary to adjust the exposure time of the camera module to match the flickering period of the ambient light flicker, so that the original image obtained can be free of screen flicker.
[0066] So far, a variety of specific implementation methods for making the working parameters of the camera module follow the ambient light flicker to eliminate the screen flicker have been described. Those skilled in the art can select the specific implementation method of step S500 according to the actual situation of the camera module and the ambient light flicker.
[0067] In another aspect of the present invention, the MR device also includes a rendering module 500, which outputs a virtual digital image at a rendering frequency. The image fusion module 300 fuses the digitized real image generated by the camera module 100 and the virtual digital image output by the rendering module 500 to enhance and render the image acquired by the camera module 100, and outputs the mixed target image for display.
[0068] Generally speaking, the rendering frequency of the rendering module 500 is consistent with the specific working frequency of the display module. In this case, the target image is obtained by fusing the intermediate image output by the image adaptation module 210 and the rendered image.
[0069] However, in some special cases, the rendering frequency may be inconsistent with the specific working frequency, resulting in a mismatch during image fusion.
[0070] To this end, in another embodiment, in response to the existence of a specific rendering frequency in the rendering module, and the specific rendering frequency does not match the specific operating frequency, the rendering module is controlled to generate a rendering image at the specific rendering frequency, and an intermediate rendering image corresponding to the specific operating frequency is generated based on the rendering image of the specific rendering frequency through the image adaptation module 220, so that the intermediate image output by the image adaptation module 210 and the intermediate rendering image output by the image adaptation module 220 can be fused at the specific operating frequency through the image fusion module 300 to obtain the target image.
[0071] The specific implementation of the image adaptation module 220 may refer to the specific implementation of the image adaptation module 210 , which will not be described in detail here.
[0072] So far, the specific implementation method of the method for eliminating the flicker of the MR device image provided by one aspect of the present invention has been described. Specifically, when the optimal frame rate of each module of the MR device is higher than the flicker frequency of the ambient light and when there is only one flicker frequency that reaches the flicker threshold in the entire shooting field of view of the ambient light, the frame rate of the camera and the captured image processing is controlled to be maintained ≤ the ambient light frequency, and the rendering and display frame rate can exceed the ambient light frequency. When displaying the camera image, the camera image is subjected to inter-frame interpolation or time warping and other techniques to obtain the image at the desired display time point. The optimal frame rate of each module of the MR device is higher than the flicker frequency of the ambient light because there are a large number of moving objects in the device rendering or shooting scene, and the frame rate of each module needs to be increased to reduce delay and display dynamic smear, or it may be because the device is a wearable head display, and the display content needs to be updated in real time following the movement of the head to improve the display realism and reduce the user's discomfort and health risks caused by factors such as delay and display dynamic smear.
[0073] According to one aspect of the present invention, the method for eliminating image flicker of an MR device can solve the problem of screen flicker or instability caused by changes in ambient light brightness and mixing of camera, rendering, and display frequencies by adjusting the shooting parameters of the camera module and generating an intermediate image based on the original image of the camera module. The present invention provides a solution for camera and display devices to adopt any refresh rate, which can dynamically adapt to changes in ambient light on the one hand, and on the other hand, provides the possibility of increasing the refresh rate of the display device, which lays the foundation for further improving the performance of mixed reality.
[0074] Another aspect of the present invention provides an MR device, such as Figure 1 As shown, the MR device at least includes a camera module 100 and a display module 400, and the MR device also includes at least one processor; and a memory coupled to the at least one processor, the memory including instructions stored therein, the instructions when executed by the at least one processor, causing the MR device to execute the method for eliminating MR device image flicker as described in any embodiment of the present invention. It should be noted that the at least one processor mentioned above may include the processors in the camera module 100 and the display module 400.
[0075] Another aspect of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for eliminating MR device image flicker as described in any of the above embodiments. For details, please refer to the above description, which will not be repeated here. In addition, it can be understood that the above-mentioned computer-readable storage medium can also be in the form of a system, that is, including multiple computer-readable storage sub-mediums, so as to jointly implement the steps of the method for eliminating MR device image flicker described above through multiple computer-readable storage media.
[0076] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.
[0077] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.
[0078] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0079] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. However, it should be understood that the scope of protection of the present invention shall be subject to the appended claims and shall not be limited to the specific structures and components of the above-described embodiments. Those skilled in the art may make various changes and modifications to the embodiments within the spirit and scope of the present invention, and these changes and modifications also fall within the scope of protection of the present invention.
Claims
1. A method for eliminating image flicker of an MR device, wherein the MR device comprises at least a camera module and a display module, characterized in that: The method comprises: In response to the presence of ambient light flicker that triggers human eye perception, determining a flicker frequency of the ambient light flicker; Determining whether the display module has a specific operating frequency according to the operating state of the MR device; In response to the existence of only a single flickering frequency and the existence of the specific operating frequency greater than the flickering frequency, always maintaining the target shooting frequency of the camera module not greater than the flickering frequency, wherein when the target shooting frequency is less than the flickering frequency, making the flickering frequency a multiple of the target shooting frequency, or making the exposure time of the camera module match the flickering period of the ambient light flickering; Based on the original image acquired by the camera module at the target shooting frequency, generating an intermediate image corresponding to the specific operating frequency; and The display module is controlled to display a target image obtained based on the intermediate image at the specific operating frequency.
2. The method according to claim 1, characterized in that Based on the original image acquired by the camera module at the target shooting frequency, generating the intermediate image corresponding to the specific working frequency further includes: At a target moment corresponding to the specific operating frequency, an intermediate image at the target moment is generated based on two frames of original images before and after the target moment.
3. The method according to claim 1, characterized in that Based on the original image acquired by the camera module at the target shooting frequency, generating the intermediate image corresponding to the specific working frequency further includes: At a target moment corresponding to the specific operating frequency, based on a change rule of a predetermined number of frames of original images before the target moment, an intermediate image at the target moment is predicted based on a previous frame of original image at the target moment.
4. The method according to claim 1, characterized in that Based on the original image acquired by the camera module at the target shooting frequency, generating the intermediate image corresponding to the specific working frequency further includes: Acquiring visual field movement data of the user according to the positioning sensor of the MR device; and At a target moment corresponding to the specific operating frequency, at least one of translation, scaling and distortion processing is performed on a previous frame of the original image at the target moment based on the field of view movement data to generate an intermediate image at the target moment.
5. The method according to any one of claims 1 to 4, characterized in that Always maintaining the target shooting frequency of the camera module not greater than the flickering frequency further includes: The target shooting frequency of the camera module is always kept lower than the flickering frequency, and the target shooting frequency of the camera module is adjusted so that the flickering frequency is a multiple of the target shooting frequency, and the exposure time of the camera module is adjusted to match the flickering period of the ambient light flicker.
6. The method according to any one of claims 1 to 4, characterized in that: The MR device further includes a rendering module, and the method further includes: Determine whether the rendering module has a specific rendering frequency; wherein In response to the existence of a specific rendering frequency in the rendering module, and the specific rendering frequency does not match the specific operating frequency, controlling the rendering module to generate a rendering image at the specific rendering frequency, generating an intermediate rendering image corresponding to the specific operating frequency based on the rendering image at the specific rendering frequency, and obtaining the target image based on the intermediate rendering image; or In response to the fact that the rendering module does not have a specific rendering frequency, the rendering module is controlled to generate a rendering image at the specific operating frequency, and the target image is also obtained based on the intermediate rendering image.
7. The method according to any one of claims 1 to 4, characterized in that: Determining whether the display module has a specific operating frequency according to the operating state of the MR device further includes: According to the working mode of the MR device and the display content of the display module, it is determined whether the display module has a specific working frequency; wherein In response to the MR device being in the energy-saving mode, the display module has a specific operating frequency that is less than a first preset threshold; In response to the MR device being in a high-performance mode, the display module has a specific operating frequency greater than a second preset threshold; In response to the display content of the display module including a video source of a specific frequency, the specific operating frequency of the display module is a multiple of the specific frequency of the video source.
8. A MR device, comprising at least a camera module and a display module, characterized in that: The MR device further includes at least one processor; and A memory coupled to the at least one processor, wherein the memory contains instructions stored therein, and when the instructions are executed by the at least one processor, the MR device executes the method for eliminating MR device image flicker according to any one of claims 1-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for eliminating image flicker of an MR device according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Photographing method, terminal, and computer readable storage medium
CN107395980A
Photographing method and device, storage medium and mobile terminal
CN109120863A
Image display method and electronic equipment
CN110557626A
Image flicker elimination method and device
CN111355864A
Image processing method and device, electronic equipment and storage medium
CN113766114A
