An optical system and electronic device

Abstract

An optical system and electronic device are disclosed to simplify the complexity of software algorithm pathways in the imaging process of electronic devices. The optical system includes a camera, a display screen, and an imaging lens. The camera has a first optical distortion, and the imaging lens has a second optical distortion. The first and second optical distortions have opposite signs, and the absolute value of the sum of the first and second optical distortions is less than a distortion threshold. By configuring the distortions of the camera and imaging lens to have opposite signs and similar magnitudes, the distortions produced by light passing through the camera and imaging lens can cancel each other out. This eliminates the need for anti-distortion processing algorithms, thereby simplifying the complexity of the software algorithm pathways.

Description

Technical Field

[0001] This application relates to the field of optical technology, and more particularly to an optical system and electronic device. Background Technology

[0002] Virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) are common technologies used to present virtual worlds to users, and they have seen rapid development in recent years. AR, MR, and XR all rely on VR as a foundation. Currently, VR and other technologies are often combined with video see-through (VST) technology. For example, in MR scenarios, VR devices can capture real-world information in real time using cameras, allowing users to still perceive the surrounding real world while wearing VR devices. This gives users the feeling of interacting with the real world through a screen, enhancing their virtual reality experience.

[0003] However, existing VR devices, after acquiring real-world images captured by a camera, typically need to send them to upper-level algorithm modules for a series of processing steps, including distortion correction and reprojection, before presenting the processed image to the user. This approach involves a long software algorithm path, resulting in high latency, power consumption, and workload from motion to photon (MTP) (i.e., from when the user turns their head to when they see the image after turning their head), which is detrimental to improving the user's VR experience.

[0004] Therefore, further research is needed on the imaging processing of electronic devices (such as VR devices). Summary of the Invention

[0005] This application provides an optical system and electronic device to simplify the complexity of the software algorithm path in the imaging process of electronic devices, reduce the latency, load and power consumption of MTP, and improve the user's VST experience.

[0006] In a first aspect, this application provides an optical system comprising a camera, a display screen, and an imaging lens. The camera has a first optical distortion, and the imaging lens has a second optical distortion. The first and second optical distortions have opposite signs, and the absolute value of the sum of the first and second optical distortions is less than a distortion threshold. The camera is used to capture a target object to obtain a first image; the display screen is used to display the first image; and the imaging lens is used to magnify the first image displayed on the display screen.

[0007] In the above scheme, by configuring the distortion of the camera and imaging lens to have opposite signs and similar magnitudes, the distortions produced by the light after passing through the camera and imaging lens can cancel each other out. This eliminates the need to introduce anti-distortion processing algorithms, thereby simplifying the complexity of the software algorithm path. In fact, the software algorithm path can be eliminated entirely, and the real-scene image captured by the camera can be directly transmitted to the display screen for display. This effectively reduces the latency, load, and power consumption of MTP, and improves the user's VST experience.

[0008] In one possible design, the first optical distortion is negative distortion, and the second optical distortion is positive distortion. For example, when designing an optical system, the camera lens can be configured as a concave lens, and the imaging lens as a convex lens, with the distortion magnitudes of these two lenses being similar in the same field of view. In this way, the positive distortion inherent in the original convex lens of the imaging lens can be directly utilized to configure the camera lens as the opposite concave lens, thereby introducing as few lenses as possible and reducing the cost and structural complexity of the optical system.

[0009] In one possible design, the distortion threshold can be a value less than 15%. For example, when the distortion threshold is configured to 10%, the sum of the first and second optical distortions is in the range of [-10%, 10%], thus balancing cost and imaging performance.

[0010] In one possible design, the optical system may also include a reflective component for reflecting emitted light from the camera to the target object and reflecting reflected light from the target object back to the camera, wherein the distance from the camera's emitted light to the target object after being reflected by the reflective component is equal to the distance of the user's line of sight to the target object.

[0011] In this design, by adding a reflective component to the optical system, the distance from the camera to the object after reflection is made the same as the distance from the human eye to the object. This eliminates the axial distance between the camera and the human eye in existing optical systems, ensuring that the field of view of the object captured by the camera is consistent with the field of view of the human eye, thus guaranteeing the accuracy of the depth information in the image obtained by the camera. Furthermore, since the depth information in the image directly captured by the camera is accurate, a reprojection algorithm is no longer needed before transmitting the image to the display screen, further simplifying the complexity of the software algorithm path.

[0012] In one possible design, the reflective component may include one or more reflective elements. For example, in scenarios where cost savings and structural complexity are required, the reflective component may be configured to include only one reflective element. In scenarios where the overall size of the electronic device needs to be reduced, the reflective component may be configured to include at least two reflective elements to utilize the reflection between the at least two reflective elements, thereby reducing the size of the optical system in one or more directions and achieving miniaturization of the electronic device.

[0013] In one possible design, the reflecting element can be an optical element with at least one reflective surface, such as a mirror or a prism. By using a common and low-cost mirror or prism as the reflecting element, the cost and complexity of the optical system can be reduced.

[0014] Secondly, this application provides an optical system comprising a camera, a reflective component, a display screen, and an imaging lens. The camera is used to emit a first light ray and receive a second light ray reflected back from the reflective component, and to process the second light ray to obtain a first image. The reflective component is used to reflect the first light ray from the camera to a target object and to reflect the second light ray from the target object back to the camera, wherein the distance from the first light ray from the camera to the target object after reflection by the reflective component is equal to the distance of the user's line of sight to the target object. The display screen is used to display the first image. The imaging lens is used to magnify the first image displayed on the display screen to obtain a second image.

[0015] In the above solution, by adding a reflective component to the optical system, the distance from the camera to the object after reflection is made the same as the distance from the human eye to the object. This eliminates the axial distance between the camera and the human eye in the existing optical system, ensuring that the field of view of the object captured by the camera is consistent with the field of view of the human eye, thus guaranteeing the accuracy of the depth information in the image obtained by the camera. Furthermore, since the depth information in the image directly captured by the camera is accurate, a reprojection algorithm is no longer needed before transmitting the image to the display screen. This simplifies the complexity of the software algorithm path, and can even eliminate the software algorithm path altogether, allowing direct transmission of the real-world image captured by the camera to the display screen. This effectively reduces the latency, load, and power consumption of the MTP (Multi-Target Projection) and improves the user's VST (Virtual Targeting) experience.

[0016] In one possible design, the camera may have a first optical distortion, and the imaging lens may have a second optical distortion. The first optical distortion and the second optical distortion have opposite signs, and the absolute value of the sum of the first optical distortion and the second optical distortion is less than the distortion threshold.

[0017] In one possible design, the first optical distortion can be negative distortion, and the second optical distortion can be positive distortion.

[0018] In one possible design, the distortion threshold can be less than 15%.

[0019] In one possible design, the reflective component may include one or more reflective elements.

[0020] In one possible design, the reflective element can be a mirror or a prism.

[0021] Thirdly, this application provides an electronic device including the optical system described in any one of the first or second aspects above.

[0022] In one possible design, the electronic device can be a near-field display device, such as VR glasses or VR helmets, or a terminal device with a display screen, such as a mobile phone, monitor, television, HUD, etc., without any specific limitation.

[0023] For details of the beneficial effects of the second and third aspects mentioned above, please refer to the technical effects that can be achieved by the corresponding design in the first aspect mentioned above, which will not be repeated here. Attached Figure Description

[0024] Figure 1 An exemplary illustration shows a field distortion curve for different wavelengths provided in an embodiment of this application;

[0025] Figure 2 This illustration shows a schematic diagram of image distortion morphology with negative and positive distortion provided in an embodiment of this application;

[0026] Figure 3 An exemplary schematic diagram of an electronic device provided in an embodiment of this application is shown;

[0027] Figure 4 An exemplary diagram illustrates a system architecture of an electronic device provided in an embodiment of this application;

[0028] Figure 5 An exemplary schematic diagram of an imaging scheme for an optical system provided in the industry is shown;

[0029] Figure 6 An exemplary schematic diagram of an imaging scheme for an optical system provided in an embodiment of this application is shown;

[0030] Figure 7 An exemplary schematic diagram of another optical system provided in an embodiment of this application is shown;

[0031] Figure 8 An exemplary schematic diagram of another optical system provided in an embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein. The same reference numerals in the figures denote the same or similar structures, and therefore repeated descriptions of them will be omitted. The terms expressing position and direction described in the embodiments of this application are illustrative based on the accompanying drawings, but changes can be made as needed, and all such changes are included within the scope of protection of this application. The accompanying drawings of the embodiments of this application are only for illustrating relative positional relationships and do not represent actual scale.

[0033] It should be noted that specific details are set forth in the following description to facilitate understanding of this application. However, the embodiments of this application can be implemented in many ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the embodiments of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0034] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0035] I. Near-eye display (NED).

[0036] NED, also known as head-mounted display or wearable display, is a type of display device that includes augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR). NED displays images close to the eyes and can create virtual images in a monocular or binocular field of view. By placing the display device within the non-visual distance of the human eye (typically less than 25 centimeters), it renders light field information to the eyes, thereby reconstructing a virtual scene in front of them.

[0037] II. Field of view (FOV).

[0038] In optical instruments, the angle between the two edges of the lens (with the lens as the vertex) representing the maximum range through which the image of the target object can pass through the lens is called the field of view (FOV). Simply put, FOV can also be understood as the angle between the edge of the display screen and the line connecting the observation point (eye). The larger the display screen, the higher the required FOV.

[0039] 3. Video see-through (VST).

[0040] VST technology is a perspective-taking technique applicable to AR, VR, MR, and XR scenarios. VST technology captures a real-time view of the surrounding environment using a camera and displays it on a screen, giving users the feeling that their eyes can directly see the real world through the display device. This enhances users' ability to interact with the real world while providing virtual imaging, and has gradually become a mainstream research direction in the NED field in recent years.

[0041] IV. Dispersion.

[0042] Dispersion refers to the phenomenon of breaking down polychromatic light into monochromatic light to form a spectrum. For example, polychromatic light can be broken down into three different colors: red (R), green (G), and blue (B). R, G, and B light have different wavelengths.

[0043] V. Distortion.

[0044] Distortion, also known as aberration, is a monochromatic optical aberration used to characterize how magnification changes within the field of view of an image at a fixed working distance. Distortion is determined by the optical design of the lens and usually exists alongside chromatic aberration. For example, please refer to... Figure 1 As shown, it illustrates field-of-view distortion curves for different wavelengths, where curve L... R This is the distortion curve corresponding to red (R) light, curve L G This is the distortion curve corresponding to green (G) light, curve L B This is the distortion curve corresponding to blue (B) light. Figure 1 As can be seen, the distortion curves of R, G, and B light do not overlap; and for any monochromatic light, the distortion increases with the increase of the field of view, that is to say, a larger field of view is more prone to distortion than a smaller field of view.

[0045] Distortion can generally be divided into negative distortion and positive distortion. Please refer to [link / reference]. Figure 2 As shown, it illustrates a schematic diagram of image distortion patterns, including negative and positive distortion. Among them, Figure 2 Image (A) shows the distortion pattern of a negatively distorted image. Points in its field of view are closer to the center, resembling a barrel shape; therefore, it can also be called barrel distortion. In contrast, Figure 2 Image (B) shows the image distortion pattern of positive distortion, where the points in the field of view are further away from the center, resembling a pillow shape. Therefore, it can also be called pillow distortion.

[0046] The degree of distortion can be measured by relative distortion, which is calculated using the following formula (1):

[0047]

[0048] Where Dist represents relative distortion; y′ represents actual image height, defined as the height of the actual ray on the image plane; and y0′ represents ideal image height, defined as the height of the reference ray on the image plane after scaling by the field of view.

[0049] In the above formula (1), the ideal image height y0′ is related to the optical design of the lens. Under normal circumstances, once the lens is manufactured, its ideal image height is fixed. The real image height y′ is related to the FOV of the lens, the optical design and manufacturing deviation of the lens. The smaller the FOV, the smaller the real image height and the less severe the relative distortion. Conversely, the larger the FOV, the larger the real image height and the more severe the relative distortion.

[0050] The following describes some of the technical features involved in the embodiments of this application.

[0051] Figure 3 This illustration shows a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device can be a NED device, such as VR glasses, VR headsets, AR glasses, AR headsets, MR glasses, MR headsets, or XR glasses, XR headsets, etc. Users can wear the NED device to play games, read, watch movies (or TV series), participate in virtual meetings, participate in video education, or video shopping, etc. In some embodiments, the electronic device can also be a terminal device with a display screen, such as a mobile phone, monitor, television, head-up display (HUD) system, etc. Figure 1 The electronic device in the illustrated embodiment is described using VR glasses as an example.

[0052] refer to Figure 3 As shown, the electronic device may include a display module 100 and a fixing component 200. The display module 100 is used to display images, and the fixing component 200 is used to support the display module 100 and fix the display module 100 in front of the user's eyes when the user wears the electronic device. For example, when the electronic device is AR glasses or VR glasses, the fixing component 200 may be a temple and a frame, wherein the frame may be connected between two temples, and the display module 100 is fixed to the frame. As another example, when the electronic device is an AR helmet or VR helmet, the fixing component 200 may be a helmet shell. The fixing component 200 may be made of metal or plastic, etc., and this application does not limit this. Furthermore, the image displayed by the display module 100 may be an image projected onto the display module 100 by a terminal device (such as a mobile phone, tablet computer, etc.), or it may be an image formed by the display module 100 itself, and this application does not limit this either.

[0053] Of course, in some other embodiments, when the electronic device is a mobile phone, the fixing component 200 can also be the phone's housing. The housing may include a mid-frame and a back cover, wherein the back cover can be fixed to one side of the mid-frame, and the display module 100 is fixed to the other side of the mid-frame opposite to the back cover. Exemplarily, the display module 100 can adopt a curved design, i.e., the display module 100 is a curved screen. In this case, the edge areas on opposite sides of the display module 100 can be bent towards the back cover.

[0054] Please refer to the above. Figure 4 As shown, Figure 4 A system architecture diagram of an electronic device provided in an embodiment of this application.

[0055] It should be understood that the illustrated electronic device is merely an example, and electronic devices may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0056] like Figure 4 As shown, electronic devices may include processors, memory, batteries, sensor modules, communication modules, cameras, eye-tracking modules, microphones, buttons, etc. The following section combines... Figure 4 A detailed introduction to each component in the electronic device.

[0057] Processors are typically used to control the overall operation of electronic devices. A processor may include one or more processing units, such as an application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units can be independent devices or integrated into one or more processors. The controller serves as the nerve center and command center of the electronic device. The controller generates operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0058] Memory is used to store instructions and data. Memory can exist independently of the processor or be configured within the processor. In some embodiments, the memory in an electronic device can be a cache memory. This memory can hold instructions or data that the processor has just used or that are used repeatedly. If the processor needs to reuse the instruction or data, it can retrieve it directly from memory, avoiding repeated accesses, reducing processor latency, and improving processing efficiency.

[0059] In some embodiments, the processor may include one or more input / output interfaces. For example, the input / output interface may include an inter-integrated circuit (I / O). 2 C) Interfaces such as Universal Asynchronous Receiver / Transmitter (UART) interface, Mobile Industry Processor Interface (MIPI) interface, General-Purpose Input / Output (GPIO) interface, Subscriber Identity Module (SIM) interface, and / or Universal Serial Bus (USB) interface, Serial Peripheral Interface (SPI) interface, etc.

[0060] I 2 The C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor may include multiple I / O pins. 2 C bus.

[0061] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be bidirectional. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect a processor and a communication module. For example, the processor communicates with a Bluetooth module in the communication module via the UART interface to implement Bluetooth functionality.

[0062] The MIPI interface can be used to connect processors to peripheral devices such as display modules or cameras.

[0063] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor to a camera, display module, communication module, sensor module, microphone, etc. The GPIO interface can also be configured as an I / O interface. 2 Interfaces include C, I2S, UART, and MIPI.

[0064] A USB interface is an interface that conforms to the USB standard specification, specifically including Mini USB, Micro USB, and USB Type-C interfaces. A USB interface can be used to connect a charger to charge the battery in an electronic device, or to transfer data between electronic devices and peripherals. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as mobile phones. A USB interface can be USB 3.0, used for compatibility with high-speed display port (DP) signal transmission, enabling the transmission of high-speed audio and video data.

[0065] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0066] Communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as Wi-Fi), Bluetooth, Global Navigation Satellite System (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. A communication module can be one or more devices integrating at least one communication processing module. The communication module can receive electromagnetic waves via an antenna, frequency-modulate and filter the electromagnetic wave signals, and send the processed signals to a processor. The communication module can also receive signals to be transmitted from the processor, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0067] In some embodiments, electronic devices can communicate with networks and other devices via wireless communication technologies. These wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0068] A camera is used to capture still images or videos. An object passes through the camera lens, generating an optical image that is projected onto a photosensitive element. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Image Signal Processor) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for further processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device may include one or more cameras.

[0069] In some embodiments, the camera captures images including real objects, the processor can fuse the images captured by the camera with virtual objects, and display the fused image through a display module.

[0070] In some embodiments, the camera may also be mounted on the lens to capture images, including the human eye, forming an eye-tracking module. The processor performs eye tracking using the images.

[0071] An optical display module is used to display images captured by a camera. The optical display module includes a display screen, which in turn includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a micro-LED, or a quantum dot light-emitting diode (QLED), etc. The display screen can be a rigid display screen, a flexible display screen, or a combination of rigid and flexible screens; the specific type is not limited.

[0072] In some embodiments, the optical display module may further include an imaging lens disposed outside the display screen. When a user wears the electronic device, the imaging lens is positioned between the display screen and the user's eyes to magnify the image displayed on the screen, allowing the user to view a larger and clearer image, thus improving the user's viewing experience.

[0073] A microphone, also known as a "voice transducer" or "microphone," is used to convert sound signals into electrical signals. Users can speak by bringing their mouth close to the microphone, inputting the sound signal into it. Electronic devices can be equipped with at least one microphone. For example, in some embodiments, electronic devices can be equipped with two microphones, which, in addition to collecting sound signals, can also perform noise reduction. Alternatively, in some embodiments, electronic devices can be equipped with three, four, or more microphones, which, in addition to collecting sound signals and noise reduction, can also identify the sound source and perform directional recording, etc.

[0074] Buttons are used by users to input commands or information. Buttons can include power buttons and volume buttons, etc. Buttons can be mechanical buttons or touch buttons. Electronic devices can receive button input and generate key signal inputs related to user settings and function control of the electronic device.

[0075] The sensor module may include one or more of the following: pressure sensor, gyroscope sensor, accelerometer sensor, distance sensor, temperature sensor, touch sensor, and bone conduction sensor. Each sensor will be described in detail below.

[0076] A pressure sensor is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor may be located on a display screen. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, or capacitive pressure sensors. A capacitive pressure sensor may consist of at least two parallel plates made of conductive material. When a force is applied to the pressure sensor, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to the display screen, the electronic device detects the intensity of the touch operation based on the pressure sensor. The electronic device may also calculate the touch location based on the detection signal from the pressure sensor. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities may correspond to different operation commands.

[0077] Gyroscope sensors can be used to determine the motion posture of electronic devices. In some embodiments, the angular velocity of the electronic device around three axes (i.e., the x, y, and z axes) can be determined using a gyroscope sensor. Gyroscope sensors can be used for image stabilization. They can also be used in motion-sensing gaming scenarios, etc.

[0078] Accelerometers can detect the magnitude of acceleration in various directions (typically three axes) of electronic devices. When the electronic device is stationary, they can detect the magnitude and direction of gravity. They can also be used to identify the attitude of electronic devices.

[0079] A distance sensor is used to measure distance. Electronic devices can measure distance using infrared or laser. In some embodiments, during a shooting scene, the electronic device can utilize a distance sensor to measure distance for rapid focusing.

[0080] A temperature sensor is used to detect temperature. In some embodiments, the electronic device uses the temperature detected by the temperature sensor to execute a temperature handling strategy. For example, when the temperature reported by the temperature sensor exceeds a threshold, the electronic device reduces the performance of a processor located near the temperature sensor to reduce power consumption and implement thermal protection. In other embodiments, when the temperature falls below another threshold, the electronic device heats the battery to prevent abnormal shutdown of the electronic device due to low temperature. In still other embodiments, when the temperature falls below yet another threshold, the electronic device boosts the battery's output voltage to prevent abnormal shutdown due to low temperature.

[0081] A touch sensor, also known as a "touch panel," is located on a display screen. The touch sensor and the display screen together form a touchscreen, also called a "touch screen." The touch sensor detects touch operations applied to or near it. It then transmits the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In some embodiments, the touch sensor may also be located on the surface of the electronic device, in a different position than the display screen.

[0082] Bone conduction sensors can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can also be integrated into headphones to form bone conduction headphones. The audio module 170 can analyze the vibration signals of the sound-vibrating bone blocks acquired by the bone conduction sensor 180M to extract voice signals and realize voice functionality. The application processor can analyze heart rate information based on the blood pressure fluctuation signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0083] although Figure 4 As not shown in the diagram, electronic devices may also include other components, such as speakers, Bluetooth devices, positioning devices, flashlights, miniature projection devices, near field communication (NFC) devices, etc., which will not be elaborated here.

[0084] In electronic devices, the optical display module and the camera together constitute the optical system. Within the optical system, the camera is also called a camera. Please refer to... Figure 5 The diagram illustrates an imaging scheme for an optical system offered by the industry. Wherein:

[0085] Figure 5 Figure (A) shows the layout and positional relationship of the various components in this optical system. Please refer to it. Figure 5 As described in (A), the optical system includes a camera 501, a display screen 502, and an imaging lens 503, with the optical axes of the camera 501, display screen 502, and imaging lens 503 coinciding. The camera 501 is positioned directly in front of the human eye, and the display screen 502 and imaging lens 503 are positioned between the camera 501 and the human eye, with the imaging lens 503 being closer to the human eye than the display screen 502.

[0086] Figure 5 The imaging process of this optical system is shown in section (B). Please refer to [the diagram]. Figure 5As shown in (B), the process includes: after the camera 501 captures the target object to obtain a real-scene image, it sends the real-scene image to the upper-level processor 510; the upper-level processor 510 processes the real-scene image using a preset image processing algorithm and transmits the processed image to the display screen 502 for display; then, the image displayed on the display screen 502 is magnified by the imaging lens 503 and the magnified image is presented to the user.

[0087] In the above imaging process, the image processing algorithms are pre-configured in the upper-level processor 510. In the VST scenario, these typically include anti-distortion processing algorithms and reprojection algorithms. Alternatively, they may include fusion algorithms, such as fusing real-world images and virtual images to obtain a hybrid virtual reality image. They may also include preprocessing algorithms, such as blurring real-world images. Other algorithms may also be included. The following section mainly introduces the anti-distortion dispersion processing algorithm and the reprojection algorithm.

[0088] Anti-distortion processing algorithm

[0089] In optical system design, due to product size and cost limitations, the display screen 502 is typically not very large. Therefore, to provide a larger field of view (FOV) and enhance user immersion, an imaging lens 503 can be placed in front of the small display screen 502 to achieve a large FOV effect by magnifying the image displayed on the display screen 502. However, this also causes image distortion after passing through the lens of the imaging lens 503, because the lens of the imaging lens 503 uses a convex lens design to magnify the image, but convex lenses themselves have pincushion distortion. Therefore, the real-scene image captured by the camera 501 needs to undergo anti-distortion processing before being transmitted to the display screen 502.

[0090] For detailed procedures on distortion correction, please refer to [link / reference]. Figure 5 As shown in (B), after the real-scene image captured by camera 501 is transmitted to processor 510, it undergoes two processing operations: In the first processing, processor 510 performs distortion compensation on the real-scene image, eliminating the distortion introduced by camera 501 to obtain a distortion-free image; in the second processing, processor 510 adds a barrel distortion to the distortion-free image, which is the opposite of the pincushion distortion of imaging lens 503. Thus, the image to be processed after the two processing operations by processor 510 is displayed on display screen 502, also presenting an image with barrel distortion. This barrel distortion "cancels out" the pincushion distortion produced by imaging lens 503, so that the human eye sees a normal image on the electronic device.

[0091] Additionally, it's important to note that distortion is a monochromatic optical aberration; that is, each monochromatic light corresponds to a specific distortion. Therefore, when adding inverse distortion to a real-world image, it's necessary to add inverse distortion separately for each monochromatic light (e.g., R-light, G-light, B-light) to ensure that the added distortion and chromatic aberration cancel out the distortion and chromatic aberration of the imaging lens 503 itself, thus simultaneously resolving both distortion and chromatic aberration issues. Because distortion and chromatic aberration coexist, in some scenarios, the anti-distortion processing algorithm is also called the anti-distortion-chromatic aberration processing algorithm.

[0092] Reprojection algorithm

[0093] In the optical system design, the camera 501 is positioned directly in front of the human eye; therefore, there is an axial distance between the camera 501 and the human eye, for example... Figure 5 The Δh is shown in (A). Due to the existence of this axial distance Δh, the depth of view captured by camera 501 (i.e., Figure 5 The h1 shown in (A) is smaller than the visual depth of the human eye (i.e., Figure 5 As shown in (A), h2), thus enabling the FOV (i.e., the field of view) captured by camera 501. Figure 5 The V1 shown in (A) is smaller than the FOV seen by the human eye (i.e., Figure 5 As shown in (A) (V2), the object captured by camera 501 appears larger in the real-world image than the object appears to the human eye in the viewing screen. Therefore, when the user turns their head, this depth difference causes the rotation speed of the real-world image captured by camera 501 to exceed the user's head rotation speed, leading to dizziness and other unpleasant experiences such as misperception of size and depth. Therefore, the real-world image captured by camera 501 needs to undergo reprojection processing before being transmitted to display screen 502, as detailed in [link to documentation]. Figure 5 As shown in (B), the real-world image captured by camera 501 is first transmitted to processor 510. Under the action of the reprojection algorithm of processor 510, the external physical world is projected onto the position of the human eye using other cameras or sensors to reconstruct the depth of the physical world, obtaining a depth map. Then, the depth information of the real-world image is adjusted by combining the depth map and displayed on display screen 502. It is then magnified by imaging lens 503 and presented to the user. In this way, by compensating for the depth difference between the real-world image captured by camera 501 and the real image seen by the user's eyes, it helps the human eye see a normal image with the same depth as the scene it sees from the electronic device.

[0094] The aforementioned optical system design, by first transmitting the real-scene image captured by the camera to the processor for anti-distortion and reprojection processing, can compensate for distortion and depth discrepancies inherent in the optical system design, helping users see a normal image. However, this solution involves a long software algorithm path; the real-scene image captured by the camera needs to undergo a lengthy algorithmic processing process before being displayed to the user. This results in high latency, power consumption, and load from the moment the user turns their head until they see the altered image (also known as motion to photon (MTP)), which is detrimental to improving the user's VST experience.

[0095] In addition, currently used reprojection algorithms all require depth reconstruction of the real physical world. However, the current reconstruction accuracy is limited. Generally, to improve reconstruction efficiency, all depth is simply assumed to be a single depth or represented by a few depths. But in reality, the objects being photographed are three-dimensional, and each object may have a unique depth at different locations. This depth reconstruction method, which uses a single depth or a few depths to represent depth, may result in a significant difference between the reconstructed depth and the true depth. This leads to certain local distortions in the reprojected image, further hindering the user's VST experience.

[0096] In view of this, embodiments of this application provide an optical system that, through the design of the components included in the optical system, enables images captured by a camera to be directly sent to a display screen for display, eliminating the need for preprocessing with anti-distortion algorithms and / or reprojection algorithms. This simplifies the complexity of the software algorithm path, reduces the latency, load, and power consumption of MTP, and improves the user's VST experience.

[0097] In view of the above-mentioned technical problems, the technical solutions provided by the embodiments of this application are described below with reference to the accompanying drawings.

[0098] In the following description of this application, "multiple" can be understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. "One or more of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, one or more of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0099] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order. For example, the "first image" mentioned below refers to an image obtained by the camera from a real scene, while the "second image" is an image magnified by the imaging lens. This does not imply that the two images have a different order, priority, or importance.

[0100] Implementation Plan 1

[0101] Please refer to Figure 6 The diagram illustrates an imaging scheme for an optical system according to an embodiment of this application. The arrangement and positional relationships of the various components in this optical system can be similar to those described above. Figure 5 The optical system shown in (A) is the same, except that in this optical system, the camera 501 has a first optical distortion, and the imaging lens 503 has a second optical distortion. The signs of the first and second optical distortions are opposite, and the absolute value of the sum of the first and second optical distortions is less than the distortion threshold. Thus, after the camera 501 captures the target object, it obtains a first image with the first optical distortion, which can be directly transmitted to the display screen 502 for display. Then, the first image displayed on the display screen 502 is magnified by the imaging lens 503. Under the action of the second optical distortion of the imaging lens 503, the first optical distortion in the first image is compensated to obtain a second image.

[0102] Specifically, when the first optical distortion and the second optical distortion have opposite signs and their sum is 0, the second image has no optical distortion; that is, the user sees a normal, distortion-free image. Conversely, when the first optical distortion and the second optical distortion have opposite signs but their sum is not 0, the second image has a third optical distortion, and the absolute value of the third optical distortion is less than the absolute value of the second optical distortion. In other words, although the user sees a distorted image, the degree of distortion is less than the degree of distortion introduced by the imaging lens 503 itself.

[0103] For example, optical distortion can be characterized by relative distortion Dist (see formula (1) above), and the absolute value of optical distortion is also the absolute value of relative distortion Dist, i.e.

[0104] For example, the distortion threshold can be a threshold configured by those skilled in the art based on experience, or it can be a threshold obtained through experimental verification that ensures the rendering effect of the second image meets the requirements. For instance, in one example, when optical distortion is characterized by relative distortion Dist, the distortion threshold can be configured to a value less than 15%. For example, experimental verification has shown that when the distortion threshold is configured to 10%, the difference between the absolute values ​​of the first and second optical distortions is within the range of [-10%, 10%], thus balancing cost and imaging effect.

[0105] For example, taking optical distortion as characterized by relative distortion Dist and a distortion threshold of 10%, the relationship between the first optical distortion and the second optical distortion can be seen in Table 1 below:

[0106] Table 1

[0107] <![CDATA[First optical distortion Dist1]]> <![CDATA[Second optical distortion Dist2]]> [-a-10%, -a+10%] a [a-10%, a+10%] -a

[0108] Where a is a non-negative real number.

[0109] As shown in Table 1, when designing an optical system:

[0110] If the second optical distortion Dist2 introduced by the imaging lens 503 is a positive distortion a, then the camera 501 can be configured to have negative distortion, and the value of the negative distortion is kept within 10% before and after the positive distortion a. For example, if the imaging lens 503 introduces a 30% second optical distortion Dist2, then the first optical distortion Dist1 introduced by the camera 501 can be configured as any negative distortion within the range of [-40%, -20%], such as -25%. In this way, after the mutual compensation between the camera 501 and the imaging lens 503, the second image finally presented to the user has only 5% relative distortion.

[0111] Conversely, if the second optical distortion Dist2 introduced by the imaging lens 503 is a negative distortion -a, then the camera 501 can be configured to have positive distortion, and the values ​​of positive and negative distortion are kept within 10% before and after the negative distortion -a. For example, if the imaging lens 503 introduces a second optical distortion Dist2 of -30%, then the first optical distortion Dist1 introduced by the camera 501 can be configured as any positive distortion within the range of [20%, 40%], such as 30%. In this way, after the camera 501 and the imaging lens 503 compensate for each other, the second image finally presented to the user does not have relative distortion.

[0112] It should be noted that the distortion values ​​mentioned here are all based on ideal conditions. In actual operation, there may be some process errors. However, it should be understood that any scheme with distortion values ​​within a certain error range from the distortion values ​​given here is within the protection scope of the embodiments of this application. The embodiments of this application do not make specific limitations on this.

[0113] For example, considering that the imaging lens 503 uses a convex lens to achieve image magnification, and that a convex lens has pincushion distortion (i.e., positive distortion), the optical system can be designed with first optical distortion as barrel distortion (i.e., negative distortion) and second optical distortion as pincushion distortion. For instance, the lens of the camera 501 can be configured as a concave lens, and the lens of the imaging lens 503 as a convex lens, with the relative distortion values ​​of these two lenses being similar in the same field of view (absolute difference controlled within 10%). Thus, the first image captured by the camera 501 inherently carries barrel distortion. After being displayed on the display screen 502, this first image is transmitted to the imaging lens 503, where it is magnified by the concave lens. Simultaneously, the pincushion distortion of the concave lens cancels out the barrel distortion introduced by the camera 501. Therefore, when the magnified image reaches the human eye, it can be restored to a distortion-free or minimally distorted image.

[0114] It should be noted that the above example directly utilizes the inherent positive distortion of the convex lens of the imaging lens 503 to configure the lens of the camera 501 as a concave lens, thereby introducing as few lenses as possible into the optical system. However, in other embodiments, additional lenses can be introduced, such as by setting other lenses besides the convex lens in the imaging lens 503, so that the combined effect of the multiple lenses in the imaging lens 503 results in negative distortion, while the lens of the camera 501 is configured to have positive distortion; or by configuring multiple lenses in the camera 501 so that the combined effect of the multiple lenses results in positive distortion. There are many feasible implementation methods, but any scheme that allows the camera 501 to directly cancel the distortion introduced by the imaging lens 503 through the lens settings of the imaging lens 503 and the camera 501 is within the protection scope of the embodiments of this application.

[0115] Furthermore, since distortion is a monochromatic optical aberration, when configuring the camera 501 and the imaging lens 503, it is necessary to configure them separately for each monochromatic light. For example, the relative distortion corresponding to the R-light of the lens of the camera 501 and the relative distortion corresponding to the R-light of the lens of the imaging lens 503 have opposite signs and similar magnitudes; the relative distortion corresponding to the G-light of the lens of the camera 501 and the relative distortion corresponding to the G-light of the lens of the imaging lens 503 have opposite signs and similar magnitudes; and the relative distortion corresponding to the B-light of the lens of the camera 501 and the relative distortion corresponding to the B-light of the lens of the imaging lens 503 have opposite signs and similar magnitudes. This is so that while the relative distortions of the camera 501 and the imaging lens 503 cancel each other out, their chromatic aberrations also cancel each other out.

[0116] In the above implementation scheme one, by configuring the distortion of the camera and imaging lens under the same field of view to have opposite signs and similar magnitudes, the distortion generated by the light after passing through the camera and imaging lens can be basically canceled out. Thus, before transmitting the image captured by the camera to the display screen, the anti-distortion processing algorithm does not need to be introduced, thereby simplifying the complexity of the software algorithm path, or even eliminating the software algorithm path, and directly transmitting the real-scene image captured by the camera to the display screen for display, so as to effectively reduce the latency, load and power consumption of MTP, and improve the user's VST experience.

[0117] Implementation Plan 2

[0118] Please refer to Figure 7 As shown, it illustrates a schematic diagram of another optical system provided in an embodiment of this application. Wherein, Figure 7 Image (A) shows a top view of the optical system. Figure 7 The image shown in section (B) is a right-hand view of the optical system; please refer to it as well. Figure 7 (A) and Figure 7 As shown in (B), in addition to a camera 501, a display screen 502, and an imaging lens 503, the optical system may also include a reflective component 504. When photographing a target object, the camera 501 emits a first ray of light, which is reflected by the reflective component 504 and transmitted to the target object. After being reflected by the target object, a second ray of light is formed, which is then reflected back to the camera 501 by the reflective component 504. The camera 501 generates a first image based on the received second ray of light and sends the first image to the display screen 502 for display. The image displayed on the display screen 502 is magnified by the imaging lens 503 to obtain a second image, which is then presented to the user.

[0119] In this configuration, the distance from which the light emitted by camera 501 reaches the target object after reflection by reflector 504 is equal to the distance of the user's line of sight to the target object. This can be understood as the sum of the distance any light emitted by the camera travels to reflector 504 and the distance traveled to the target object after reflection by reflector 504 equaling the distance of the user's line of sight to the target object. For example, with... Figure 7 Taking the bottom edge ray as shown in (A) as an example, the distance from the bottom edge ray emitted by the camera 501 to the reflector 504 is L1, and the distance from the reflector 504 to the target object after reflection is L2. The distance from the human eye to the target object is L3. Then, L1, L2 and L3 satisfy the following formula (2):

[0120] L3 = L1 + L2……(2)

[0121] Please continue to refer to Figure 7 As shown in (A), since the distance from the light emitted by camera 501 to the target object after reflection by reflector 504 is equal to the distance of the user's line of sight to the target object, the depth of the target object captured by camera 501 is consistent with the depth of the target object seen by the user's eye. Consequently, the field of view (FOV) of the target object captured by camera 501 is also consistent with the field of view (FOV) of the target object seen by the user's eye, for example, both being equal. Figure 7 V is shown in (A). Thus, the image obtained by camera 501 when capturing the target object is the same as the image seen by the user when viewing the target object. The image captured by camera 501 can be directly transmitted to display screen 503 for display without needing to be re-projected by the processor.

[0122] For example, reflective component 504 may include one or more reflective elements. For instance:

[0123] In some embodiments, such as Figure 7 As shown in (A), the reflective assembly 504 may consist of only one reflective element, the center of which is located on the optical axis of the display screen 502 and the imaging lens 503, and the camera 501 is disposed inside the device housing shown below. In this way, the light emitted by the camera 501 is directly reflected to the target object by the reflective element, and the reflection direction coincides with the line of sight of the human eye towards the target object. Achieving the same shooting depth of the camera 501 as the viewing depth of the human eye using only one reflective element minimizes the introduction of additional components, reduces the complexity of the optical system, and saves costs.

[0124] In some embodiments, the reflective component 504 may include at least two reflective elements, such as those described above. Figure 8The illustrated optical system includes a reflective assembly 504 comprising two reflective elements, namely reflective element 5041 and reflective element 5042. The center point of reflective element 5042 is located on the optical axis of the display screen 502 and the imaging lens 503. The camera 501 is disposed inside the device housing in the lower right corner of the illustration, and the center point of reflective element 5041 is located on the optical axis of camera 501. Thus, light emitted from camera 501 is first reflected by reflective element 5041 to reflective element 5042, and then reflected by reflective element 5042 to the target object, with the reflection direction of reflective element 5042 coinciding with the line of sight of the human eye towards the target object. Compared to... Figure 7 For the optical system shown in (A), Figure 8 The optical system in the diagram is shorter in the vertical direction, meaning that the camera's shooting depth is consistent with the human eye's viewing depth through at least two reflective elements. This also helps to reduce the overall size of electronic devices, make the internal component layout more compact, and facilitate the miniaturization of electronic devices.

[0125] For example, the aforementioned reflective element can be any component capable of reflecting input light, such as an optical element with at least one reflective surface, like a mirror or a prism. The mirror can be a plane mirror, a spherical mirror, or an aspherical mirror, etc. A prism, also known as a faceted mirror, can be a triangular prism, a right-angle prism, or a pentagonal prism, etc.

[0126] In the second implementation scheme described above, by adding a reflective component to the optical system, the distance from the camera to the object after reflection by the reflective component becomes the same as the distance from the human eye to the object. This eliminates the axial distance between the camera and the human eye in the existing optical system, ensuring that the viewing angle of the object captured by the camera is consistent with the viewing angle of the human eye, thus guaranteeing the accuracy of the depth information in the image obtained by the camera. Furthermore, since the depth information in the image directly captured by the camera is accurate, a reprojection algorithm is no longer needed before transmitting the image to the display screen. This simplifies the complexity of the software algorithm path, and may even eliminate the software algorithm path altogether, allowing the real-world image captured by the camera to be directly transmitted to the display screen for display. This effectively reduces the latency, load, and power consumption of MTP, improving the user's VST experience.

[0127] It should be noted that the above implementation scheme one can simplify the anti-distortion processing algorithm in the original algorithm path, while implementation scheme two can simplify the reprojection algorithm in the original algorithm path. The above implementation scheme one and implementation scheme two are only introduced from the perspective of how to simplify one of the algorithms. However, in actual operation, these two implementation schemes can also be combined. For example, in one possible combination scheme, not only is the first optical distortion configured in the optical system opposite to the second optical distortion of the imaging lens, but a reflection component is also added to the optical system. In this way, both the anti-distortion processing algorithm and the reprojection algorithm in the original algorithm path can be simplified, further reducing the complexity of the software algorithm path. Or even if the software algorithm path is deleted, the accuracy of the depth information of the displayed image can be ensured by relying on the design of the optical system itself, while displaying the image without distortion or with minimal distortion.

[0128] In addition, this application also provides an electronic device including the aforementioned imaging system, such as including the aforementioned imaging system and a housing, with the imaging system encapsulated in the housing.

[0129] For example, the electronic device can be a NED device, such as VR glasses or VR headset, or a terminal device with a display screen, such as a mobile phone, monitor, TV, HUD, etc., without any specific limitation.

[0130] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An optical system, characterized in that, The device includes a camera, a display screen, and an imaging lens. The camera has a first optical distortion, and the imaging lens has a second optical distortion. The first optical distortion and the second optical distortion have opposite signs, and the absolute value of the sum of the first optical distortion and the second optical distortion is less than a distortion threshold. The camera is used to capture the target object and obtain a first image; The display screen is used to display the first image; The imaging lens is used to magnify the first image displayed on the display screen.

2. The optical system as described in claim 1, characterized in that, It also includes a reflection component; The reflective component is used to reflect emitted light from the camera to the target object, and to reflect reflected light from the target object back to the camera; Wherein, the distance from the camera's emitted light rays to the target object after being reflected by the reflective component is equal to the distance of the user's line of sight to the target object.

3. The optical system as described in claim 1 or 2, characterized in that, The first optical distortion is negative distortion, and the second optical distortion is positive distortion.

4. The optical system as described in claim 1 or 2, characterized in that, The distortion threshold is less than 15%.

5. The optical system as described in claim 2, characterized in that, The reflective assembly includes one or more reflective elements.

6. The optical system as claimed in claim 5, characterized in that, The reflective element is a mirror or a prism.

7. An electronic device, characterized in that, Includes the optical system as described in any one of claims 1 to 6.

Images