Mixed reality device, method for adjusting brightness of the same, and mixed reality surgery navigation system

Abstract

The application is suitable for the field of mixed reality surgery navigation, and provides a mixed reality device and a brightness adjusting method thereof, and a mixed reality surgery navigation system, which comprises a manual adjusting mode and an adaptive adjusting mode. The manual adjusting mode allows a user to directly adjust the brightness parameter of the device, and the adaptive adjusting mode automatically optimizes the brightness parameter by sensing the change of ambient light in real time without manual intervention. After the user selects a target adjusting mode, the processor of the device will execute corresponding brightness adjusting operation according to the mode type. The design realizes double advantages: on the one hand, the flexibility of manual adjustment is retained to meet individual needs; on the other hand, the adaptive mode adjusts the brightness intelligently, which not only eliminates the distraction risk of manual operation by doctors during surgery, but also ensures that the virtual image is always clear and visible under various lighting conditions. This dynamic adjusting mechanism significantly improves the concentration, smoothness and accuracy of surgical operation.

Description

Technical Field

[0001] This application belongs to the field of mixed reality surgical navigation technology, and particularly relates to a mixed reality device and its brightness adjustment method, and a mixed reality surgical navigation system. Background Technology

[0002] Mixed Reality (MR) technology is a technology that blends virtual information with the real world, and it has been widely used in fields such as medicine, education, and industry in recent years. Particularly in the medical field, devices with MR capabilities are gradually becoming important tools for surgical assistance. However, current display technologies for MR devices still have significant limitations in their application in special environments such as operating rooms.

[0003] Operating room environments typically have complex lighting conditions. As ambient light changes or the surgeon's operational needs evolve, the required brightness of the observed virtual image and the actual object dynamically change. For example, under high-brightness lighting conditions such as operating room lamps, the virtual image brightness may be difficult to clearly display due to excessive ambient light. To address this issue, current technologies typically involve the surgeon manually adjusting the ambient light transmittance of the mixed reality device's lenses to reduce interference from ambient light on the virtual image. However, this manual adjustment method has significant drawbacks: firstly, surgeons require a high degree of concentration during surgery, and manual adjustment can easily distract them, increasing surgical risks; secondly, manual adjustment cannot respond to changes in ambient light in real time, resulting in low matching efficiency between the virtual image brightness and the actual object brightness, affecting the accuracy and smoothness of the surgical procedure. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and storage medium for adjusting the brightness of a mixed reality device.

[0005] In a first aspect, embodiments of this application provide a brightness adjustment method for a mixed reality device, comprising: in response to a mode selection operation triggered by a user, determining a target adjustment mode from a plurality of preset brightness adjustment modes; wherein the plurality of brightness adjustment modes include a manual adjustment mode and an adaptive adjustment mode; and adjusting the brightness parameters of the mixed reality device based on the target adjustment mode.

[0006] In one embodiment, the adaptive adjustment mode includes an ambient light sensing mode and a camera tracking mode.

[0007] In one embodiment, the target adjustment mode is an ambient light sensing mode; based on the target adjustment mode, the brightness parameters of the mixed reality device are adjusted, including: acquiring the output data of the ambient light sensor of the mixed reality device; determining the ambient light brightness within the sensing range of the ambient light sensor based on the output data; and adjusting the brightness parameters of the mixed reality device based on the ambient light brightness.

[0008] In one embodiment, the target adjustment mode is a camera tracking mode; based on the target adjustment mode, the brightness parameters of the mixed reality device are adjusted, including: acquiring the output data of the camera module of the mixed reality device; determining the ambient light brightness within the target imaging range of the camera module according to the output data; and adjusting the brightness parameters of the mixed reality device according to the ambient light brightness.

[0009] In one embodiment, the target imaging range is: the global imaging range of the camera module; or, at least one range within the global imaging range where the brightness meets a preset brightness condition; or, at least one range determined by the user within the global imaging range; or, the user's binocular focusing range.

[0010] In one embodiment, the brightness parameters include ambient light transmittance and virtual image brightness; adjusting the brightness parameters of the mixed reality device according to the ambient light brightness includes: obtaining the virtual image brightness and a preset brightness contrast of the mixed reality device; adjusting the brightness parameters of the mixed reality device according to the ambient light brightness, virtual image brightness, and preset brightness contrast using a preset rule; wherein the preset rule is: first adjust the ambient light transmittance, and if the ratio of the virtual and real image brightness at eye level obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast, then adjust the virtual image brightness.

[0011] In one embodiment, the ambient light transmittance has a first adjustment range with two first endpoint values, and the virtual image brightness has a second adjustment range with two second endpoint values. The brightness parameters of the mixed reality device are adjusted according to a preset rule based on the ambient light brightness and a preset brightness contrast ratio, including: determining a transmittance target value based on the ambient light brightness, virtual image brightness, and preset brightness contrast ratio; if the transmittance target value is within the first adjustment range, adjusting the ambient light transmittance to the transmittance target value; otherwise, adjusting the ambient light transmittance to the first target endpoint value; wherein the first target endpoint value is the first endpoint value with the smallest difference from the transmittance target value among the two first endpoint values; if the virtual-to-real image brightness ratio obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast ratio, a brightness target value is determined based on the ambient light brightness, the adjusted ambient light transmittance, and the preset brightness contrast ratio; if the brightness target value is within the second adjustment range, adjusting the virtual image brightness to the brightness target value; otherwise, adjusting the virtual image brightness to the second target endpoint value; wherein the second target endpoint value is the second endpoint value with the smallest difference from the brightness target value among the two second endpoint values.

[0012] In one embodiment, the camera tracking mode includes an automatic feature tracking mode, and the target adjustment mode is the automatic feature tracking mode. Based on the target adjustment mode, adjusting the brightness parameters of the mixed reality device includes: acquiring and displaying a first environmental image output by the camera module of the mixed reality device; in response to a user-triggered feature selection operation, determining at least one feature and the weight of each feature from the first environmental image; acquiring a second environmental image output by the camera module; determining the brightness and weight of each feature based on the second environmental image; determining the ambient light brightness based on the brightness and weight of each feature; and adjusting the brightness parameters of the mixed reality device based on the ambient light brightness.

[0013] In one embodiment, the weights of each feature decrease sequentially according to the selection order of the features, and the sum of the weights of each feature equals 1.

[0014] In one embodiment, the camera tracking mode includes an automatic focus tracking mode, and the target adjustment mode is an automatic focus tracking mode; based on the target adjustment mode, the brightness parameters of the mixed reality device are adjusted, including: acquiring the user's binocular focus points and a third ambient image output by the camera module of the mixed reality device; determining the focus range and the brightness of the focus range from the third ambient image based on the binocular focus points, and using the brightness of the focus range as the ambient light brightness; and adjusting the brightness parameters of the mixed reality device based on the ambient light brightness.

[0015] Secondly, embodiments of this application provide a mixed reality device, including a processor, a display module, optical lenses, an ambient light sensor, and a camera module; the processor is used to perform: in response to a mode selection operation triggered by a user, determining a target adjustment mode from a set of preset brightness adjustment modes; wherein the set of brightness adjustment modes includes a manual adjustment mode and an adaptive adjustment mode; and adjusting the brightness parameters of the mixed reality device based on the target adjustment mode.

[0016] Thirdly, embodiments of this application provide a mixed reality surgical navigation system, including the mixed reality device described in the second aspect.

[0017] Fourthly, embodiments of this application provide a brightness adjustment device for a mixed reality device, the device comprising:

[0018] The mode determination module is used to determine the target adjustment mode from a variety of preset brightness adjustment modes in response to a mode selection operation triggered by the user; the various brightness adjustment modes include manual adjustment mode and adaptive adjustment mode.

[0019] The brightness adjustment module is used to adjust the brightness parameters of the mixed reality device based on the target adjustment mode.

[0020] Fifthly, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method steps described in any one of the first aspects above.

[0021] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0022] In a seventh aspect, embodiments of this application provide a computer program product that, when run on a computer device, enables the computer device to execute the brightness adjustment method for a mixed reality device as described in any of the first aspects.

[0023] The aforementioned mixed reality device and its brightness adjustment method, along with the mixed reality surgical navigation system, support multiple brightness adjustment modes, including manual and adaptive modes. The manual adjustment mode allows users to directly adjust the device's brightness parameters, while the adaptive mode automatically optimizes brightness parameters by sensing changes in ambient light in real time, requiring no manual intervention. After the user selects a target adjustment mode, the device's processor executes the corresponding brightness adjustment operation based on the mode type. This design achieves a dual advantage: on the one hand, it retains the flexibility of manual adjustment to meet personalized needs; on the other hand, the adaptive mode, through intelligent brightness adjustment, eliminates the risk of distraction for doctors during surgery while ensuring that the virtual image remains clearly visible under various lighting conditions. This dynamic adjustment mechanism significantly improves the focus, smoothness, and accuracy of surgical operations, providing a more reliable solution for the application of mixed reality technology in medical scenarios. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of the brightness adjustment method provided in this embodiment;

[0026] Figure 2 This is a flowchart of the brightness adjustment method based on an ambient light sensor provided in this embodiment;

[0027] Figure 3 This is a flowchart of the camera-based brightness adjustment method provided in this embodiment;

[0028] Figure 4This is a flowchart of the brightness parameter adjustment logic provided in this embodiment;

[0029] Figure 5 A flowchart illustrating the specific logic for adjusting the brightness parameters provided in this embodiment;

[0030] Figure 6 This is a flowchart of the automatic feature tracking mode method provided in this embodiment;

[0031] Figure 7 This is a flowchart of the automatic focus tracking mode method provided in this embodiment;

[0032] Figure 8 This is a functional component architecture diagram of the mixed reality device provided in this embodiment;

[0033] Figure 9 This is a functional component architecture diagram of the mixed reality surgical navigation system provided in this embodiment;

[0034] Figure 10 This is a block diagram of the brightness adjustment device provided in this embodiment. Detailed Implementation

[0035] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] This embodiment provides a brightness adjustment method for a mixed reality device. The mixed reality device can be a head-mounted display device, such as augmented reality (AR) glasses or AR helmets. Mixed reality devices typically include components such as a processor, display module, optical lenses, ambient light sensor, and camera module. This method provides multiple brightness adjustment modes, allowing users to select the appropriate brightness adjustment method according to different usage scenarios. Figure 1 This is a flowchart of the brightness adjustment method provided in this embodiment. The execution entity of this method can be the processor of a mixed reality device, such as... Figure 1 As shown, the method includes:

[0042] Step 101: In response to the user-triggered mode selection operation, determine the target adjustment mode from a variety of preset brightness adjustment modes.

[0043] In this step, the mixed reality device has several preset brightness adjustment modes, including manual and adaptive modes. Users can trigger mode selection through physical buttons, touch panels, voice commands, or gestures on the device. For example, users can slide to select different brightness adjustment modes using the side touchpad of the mixed reality device, or select the target adjustment mode using voice commands such as "switch to manual mode".

[0044] When a user triggers a mode selection operation, the processor of the mixed reality device receives the operation instruction and determines the target adjustment mode selected by the user from a variety of preset brightness adjustment modes. For example, if the user swipes up on the touchpad and stays on the "Manual Adjustment Mode" option, the processor will determine that the target adjustment mode is manual adjustment mode.

[0045] The manual adjustment mode allows users to directly adjust the brightness parameters of the mixed reality device, such as ambient light transmittance and virtual image brightness. Ambient light transmittance refers to the proportion of real ambient light allowed to pass through the optical lenses of the mixed reality device; virtual image refers to the virtual content projected by the display module of the mixed reality device for the user to view; and virtual image brightness refers to the brightness of the virtual content at the imaging location. The adaptive adjustment mode automatically adjusts the brightness parameters according to environmental conditions, without requiring manual intervention from the user.

[0046] Step 102: Adjust the brightness parameters of the mixed reality device based on the target adjustment mode.

[0047] After determining the target adjustment mode, the processor of the mixed reality device will perform corresponding brightness adjustment operations according to different adjustment modes.

[0048] If the target adjustment mode is manual, the processor will respond to the user's brightness adjustment command and directly adjust the brightness parameters of the mixed reality device. For example, the user can increase or decrease the ambient light transmittance and virtual image brightness using the brightness adjustment slider or buttons on the device.

[0049] If the target adjustment mode is adaptive adjustment mode, the processor will automatically adjust the brightness parameters according to environmental conditions. Adaptive adjustment mode can automatically adjust the brightness parameters of the mixed reality device based on factors such as ambient light and user activity to provide the best visual experience.

[0050] The brightness adjustment method for a mixed reality device provided in this embodiment includes a manual adjustment mode and an adaptive adjustment mode. The manual adjustment mode allows users to directly adjust the device's brightness parameters, while the adaptive adjustment mode automatically optimizes the brightness parameters by sensing changes in ambient light in real time, requiring no manual intervention. After the user selects a target adjustment mode, the device's processor will execute the corresponding brightness adjustment operation according to the mode type. This design achieves a dual advantage: on the one hand, it retains the flexibility of manual adjustment to meet personalized needs; on the other hand, the adaptive mode, through intelligent brightness adjustment, eliminates the risk of distraction for doctors during surgery while ensuring that the virtual image remains clearly visible under various lighting conditions. This dynamic adjustment mechanism significantly improves the focus, smoothness, and accuracy of surgical operations, providing a more reliable solution for the application of mixed reality technology in medical scenarios.

[0051] Based on the above embodiments, this embodiment further refines the types of adaptive adjustment modes, including ambient light sensing mode and camera tracking mode.

[0052] The adaptive adjustment modes include ambient light sensing mode and camera tracking mode. Ambient light sensing mode uses the ambient light sensor on the mixed reality device to sense the brightness of the surrounding environment and adjusts the brightness parameters of the mixed reality device accordingly. Camera tracking mode uses the camera module on the mixed reality device to capture environmental images, analyzes the brightness information in the images, and adjusts the brightness parameters of the mixed reality device accordingly; or it uses the exposure time of the camera module to determine and evaluate the ambient light brightness and adjusts the brightness parameters of the mixed reality device accordingly.

[0053] The ambient light sensing mode is suitable for scenarios that are sensitive to changes in overall ambient light brightness, such as when switching between indoor and outdoor environments or when lighting conditions change significantly. This mode can quickly respond to changes in ambient light brightness and provide real-time brightness adjustment.

[0054] The camera tracking mode is more precise, capable of analyzing the brightness distribution within the user's field of view and adjusting the brightness of specific areas. This mode is suitable for scenes with uneven brightness distribution, such as mixed lighting environments where some areas are bright and others are dark.

[0055] Users can choose the appropriate adaptive adjustment mode based on the actual usage scenario and personal preference. For example, in scenarios where lighting conditions change frequently, users can choose the ambient light sensing mode; in scenarios where it is necessary to focus on the brightness of a specific area, users can choose the camera tracking mode.

[0056] This embodiment provides multiple adaptive adjustment modes, enabling mixed reality devices to adapt more flexibly to different usage environments and provide more precise brightness adjustment, thereby improving the user experience.

[0057] Based on the above embodiments, this embodiment describes in detail how the mixed reality device adjusts the brightness parameters when the target adjustment mode is ambient light sensing mode. Figure 2 This is a flowchart of the brightness adjustment method based on an ambient light sensor provided in this embodiment, as follows: Figure 2 As shown, the method includes:

[0058] Step 201: Acquire the output data of the ambient light sensor of the mixed reality device.

[0059] In this step, the processor of the mixed reality device acquires the output data from the ambient light sensor via an internal communication interface. The ambient light sensor, typically mounted on the outer surface of the mixed reality device, senses the brightness of the surrounding environment. It converts the sensed brightness information into an electrical signal and transmits it to the processor via the internal communication interface. The ambient light sensor can be a photodiode-type sensor, a digital light sensor, etc., and is not limited here.

[0060] The output data of an ambient light sensor can be either analog or digital. The processor will process and convert the data accordingly based on the type and output format of the ambient light sensor. For example, for an ambient light sensor that outputs an analog signal, the processor will convert the analog signal into a digital signal using an analog-to-digital converter; for an ambient light sensor that outputs a digital signal, the processor can directly read the digital value.

[0061] Step 202: Determine the ambient light brightness within the sensing range of the ambient light sensor based on the output data.

[0062] After acquiring the output data from the ambient light sensor, the processor converts the sensor output data into ambient light brightness according to a preset conversion relationship, and uses this as the basis for subsequent brightness adjustment. The conversion relationship is usually determined by the characteristics and calibration parameters of the ambient light sensor, and can be linear or non-linear.

[0063] Step 203: Adjust the brightness parameters of the mixed reality device according to the ambient light intensity.

[0064] After determining the ambient light level, the processor adjusts the brightness parameters of the mixed reality device according to a preset brightness adjustment strategy. Brightness parameters may include ambient light transmittance and virtual image brightness. The brightness adjustment strategy is typically based on considerations of human eye comfort and visual experience, and can be a set of preset mapping relationships or algorithms.

[0065] Through the above steps, this embodiment enables the mixed reality device to automatically adjust its brightness parameters based on the ambient light intensity sensed by the ambient light sensor, adapting to different lighting environments and providing a comfortable visual experience.

[0066] Based on the above embodiments, this embodiment describes in detail how the mixed reality device adjusts the brightness parameters when the target adjustment mode is camera tracking mode. Figure 3 This is a flowchart of the camera-based brightness adjustment method provided in this embodiment, as follows: Figure 3 As shown, the method includes:

[0067] Step 301: Obtain the output data of the camera module of the mixed reality device.

[0068] In this step, the processor of the mixed reality device acquires the output data from the camera module through an internal communication interface. The camera module, typically installed at the front of the mixed reality device, is capable of capturing environmental images within the user's field of view. The camera module transmits the acquired image data to the processor via the internal communication interface. The camera module may include an RGB camera, and may also include at least one of a depth camera, an infrared camera, a monocular camera, or a multi-view camera.

[0069] The camera module typically outputs digital image data, including information such as the brightness and color of the surrounding environment. The processor receives this image data and performs subsequent analysis and processing.

[0070] Step 302: Determine the ambient light intensity within the target imaging range of the camera module based on the output data.

[0071] After acquiring the output data from the camera module, the processor analyzes the image data to determine the ambient light intensity within the target imaging range. The target imaging range can be the global imaging range of the camera module or a specific area, such as the area of ​​interest to the user or an area where the brightness meets specific conditions.

[0072] The processor can analyze the brightness distribution in an image using image processing algorithms, such as brightness histogram analysis and region segmentation, and calculate the average or weighted brightness value within the target imaging range. These brightness values ​​can be obtained through statistical analysis of pixel values, such as calculating the average, median, or weighted average of pixel brightness.

[0073] For example, a processor can convert an image to grayscale and then calculate the average brightness value of all pixels within the target imaging range as an estimate of the ambient light brightness. Alternatively, the processor can perform region segmentation on the image, identify regions with different brightness levels, and calculate a weighted average brightness value based on the importance or size of the regions.

[0074] Step 303: Adjust the brightness parameters of the mixed reality device according to the ambient light intensity.

[0075] After determining the ambient light level, the processor adjusts the brightness parameters of the mixed reality device according to a preset brightness adjustment strategy. Brightness parameters may include ambient light transmittance and virtual image brightness. The brightness adjustment strategy is typically based on considerations of human eye comfort and visual experience, and can be a set of preset mapping relationships or algorithms.

[0076] In some other embodiments, the camera module's output data can also be the exposure time. The processor determines the ambient light intensity based on the camera module's exposure time using a preset algorithm, and adjusts the brightness parameters of the mixed reality device accordingly. For example, a camera module with adaptive exposure can dynamically adjust the exposure time based on the intensity of the ambient light; a longer exposure time in low light and a shorter exposure time in bright light. The processor can establish a mapping relationship between exposure time and ambient light intensity, determining the ambient light intensity based on the exposure time. This method requires less computation and has lower latency, making it suitable for scenarios requiring high-speed response. It can adjust brightness immediately, avoiding visual discomfort.

[0077] In some other embodiments, the camera module's output data can also be an ambient image and exposure time. The processor determines the ambient light brightness based on the ambient image and exposure time using a preset algorithm, and adjusts the brightness parameters of the mixed reality device accordingly. This approach can more accurately reproduce the brightness of the real environment, avoid misjudgments caused by a single exposure time, and provide a more stable brightness adjustment effect.

[0078] Unlike ambient light sensing mode, camera tracking mode provides more refined brightness information, allowing for targeted brightness adjustment in specific areas. For example, if there are areas with significant differences in brightness within the user's field of vision, the processor can employ different brightness adjustment strategies based on the brightness of different areas, providing more precise brightness control.

[0079] Through the above steps, this embodiment enables the mixed reality device to analyze the ambient light brightness within the target imaging range based on the environmental image captured by the camera module, and adjust the brightness parameters accordingly to provide more precise brightness adjustment and adapt to complex lighting environments.

[0080] Based on the above embodiments, this embodiment describes in detail a method for determining the target imaging range in camera tracking mode.

[0081] In camera tracking mode, the selection of the target imaging range has a significant impact on the accuracy of brightness adjustment. This embodiment provides several methods for determining the target imaging range, enabling mixed reality devices to select the appropriate target imaging range according to different usage scenarios and user needs.

[0082] Method 1: Use the global imaging range of the camera module as the target imaging range.

[0083] In this method, the processor uses the entire image captured by the camera module as the target imaging area, analyzes the brightness distribution of the entire image, and calculates a global average brightness value or a weighted brightness value. This method is suitable for scenes with relatively uniform ambient lighting and can provide overall ambient brightness information.

[0084] For example, a processor can calculate the average brightness value of the entire image as an estimate of the ambient light brightness. Alternatively, the processor can segment the image into regions, identify regions of different brightness, and calculate a weighted average brightness value based on the importance or size of the regions.

[0085] In a surgical scenario, the user is a doctor. The real environment that the doctor is concerned with and the virtual image projected by the mixed reality device that the doctor sees are located in the area illuminated by the shadowless lamp. Therefore, this area is of higher importance and can be given a higher weight when calculating the weighted average brightness value.

[0086] Method 2: Use at least one range whose brightness meets the preset brightness conditions as the target imaging range.

[0087] In this method, the processor performs brightness analysis on the images captured by the camera module, identifies regions whose brightness meets specific conditions, and uses these regions as the target imaging range. The preset brightness conditions can be brightness thresholds, brightness ranges, or brightness change rates, etc.

[0088] For example, the processor can identify areas in an image whose brightness exceeds a certain threshold, or whose brightness falls within a certain range, or whose brightness varies significantly, and use these areas as the target imaging range. This method is suitable for scenes with uneven ambient lighting and can focus on areas with specific brightness conditions.

[0089] In a surgical scenario, the user is a doctor. The real environment that the doctor is concerned with and the virtual image projected by the mixed reality device that the doctor sees are located in the area illuminated by the shadowless lamp, and the brightness of this area is much higher than that of other areas. Therefore, the processor can identify this area based on the image and use it as the target imaging range.

[0090] Method 3: Use at least one range determined by the user within the global imaging range as the target imaging range.

[0091] In this method, users can use interactive methods, such as clicking, selecting, or voice commands, to determine one or more regions of interest within the global imaging range of the camera module, and use these regions as the target imaging range.

[0092] For example, users can use the interactive interface of a mixed reality device to click or select a specific area in an image, or use voice commands such as "focus on the lower right area" to determine the target imaging range. The processor will determine the corresponding target imaging range based on the user's interaction and analyze the brightness information within that range. This method is suitable for scenarios where users need to focus on the brightness of a specific area, providing customized brightness adjustment.

[0093] In a surgical scenario, the user is a doctor. The real environment that the doctor is concerned with and the virtual image projected by the mixed reality device that the doctor sees are in the area illuminated by the shadowless lamp. The brightness of this area is much higher than that of other areas. Therefore, the doctor can select this area on the image through gesture interaction and use it as the target imaging range.

[0094] In another surgical scenario, multiple functional components such as a reference target, guide, and instruments are involved. Virtual images (e.g., virtual models of the target, guide, and instruments; relevant parameter information for the target, guide, and instruments, such as the angle of the instrument and the distance between the instrument tip and the target) need to be projected onto each component. This ensures that the virtual image (e.g., the virtual model) seen by the surgeon overlaps with the real component in the environment, or that the virtual image (e.g., the relevant parameter information) is located near the real component in the environment. For this scenario, the surgeon can use gestures to select the area containing each component on the image and treat these areas as the entire target imaging range.

[0095] Method 4: Use the user's binocular focusing range as the target imaging range.

[0096] In this method, the mixed reality device uses eye-tracking technology to identify the user's eye focus points and uses these focus points as the center to determine a focal range as the target imaging range.

[0097] For example, mixed reality devices can use a built-in eye tracker to identify the user's pupil position and gaze direction, determining the user's focal point. Then, a fixed-size area, or an area dynamically adjusted based on the focusing distance, is defined as the target imaging range, centered on this focal point. This method is suitable for scenarios where the user's gaze frequently changes, providing dynamic brightness adjustment that follows the user's gaze.

[0098] In one surgical scenario, the position of the virtual image seen by the doctor through the mixed reality device changes with the doctor's line of sight. For this scenario, an eye tracker can be used to track the doctor's eye focus point to determine the target imaging range.

[0099] This embodiment provides multiple methods for determining the target imaging range, enabling mixed reality devices to select appropriate target imaging ranges based on different usage scenarios and user needs, thus providing more precise and personalized brightness adjustment.

[0100] Based on the above embodiments, this embodiment describes in detail how a mixed reality device adjusts brightness parameters according to ambient light intensity. Figure 4 The flowchart of the brightness parameter adjustment logic provided in this embodiment is as follows: Figure 4 As shown, the method includes:

[0101] Step 401: Obtain the virtual image brightness and preset brightness contrast of the mixed reality device.

[0102] In this step, the processor of the mixed reality device acquires the virtual image brightness and a preset brightness contrast ratio. Brightness contrast ratio is the ratio of the virtual image brightness to the ambient light intensity at eye level, used to measure the brightness difference between the virtual image and the ambient light. The ambient light intensity at eye level can be calculated from the ambient light intensity and ambient light transmittance; for example, it can be obtained by multiplying the ambient light intensity and ambient light transmittance.

[0103] The virtual image brightness can be the current virtual image brightness, or it can be set directly by the user (usually different from the current virtual image brightness), or it can be calculated based on the user's preferred parameters.

[0104] The preset brightness and contrast can be a fixed value, a user-defined value, or a value that is dynamically adjusted according to user preferences, usage scenarios, or device characteristics.

[0105] The processor can read the virtual image brightness or preset brightness / contrast ratio from the device's configuration file, or directly use the user-input value as the virtual image brightness or preset brightness / contrast ratio, or calculate the virtual image brightness or preset brightness / contrast ratio based on the user's preferred parameters. For example, the user can select options such as "high contrast," "medium contrast," or "low contrast" through the device's settings interface, and the processor will set the corresponding preset brightness / contrast ratio according to the user's selection.

[0106] Step 402: Adjust the brightness parameters of the mixed reality device according to the ambient light brightness, virtual image brightness, and preset brightness contrast using preset rules.

[0107] After acquiring the ambient light brightness and preset brightness contrast, the processor adjusts the brightness parameters of the mixed reality device according to preset rules. The brightness parameters include ambient light transmittance and virtual image brightness.

[0108] The preset rule is as follows: First, adjust the ambient light transmittance. If the ratio of the virtual image brightness to the ambient light brightness obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast, then adjust the virtual image brightness. The ratio of the virtual image brightness to the ambient light brightness is used to measure the brightness difference between the virtual image and the ambient light.

[0109] The specific adjustment process is as follows:

[0110] First, the processor adjusts the ambient light transmittance of the mixed reality device based on the ambient light brightness, the virtual image brightness, and the preset brightness contrast. Ambient light transmittance determines the proportion of external ambient light that passes through the device's lenses and enters the user's eyes. By adjusting the ambient light transmittance, the brightness of ambient light entering the eye can be controlled, thereby affecting the brightness ratio of the virtual and real images entering the eye.

[0111] For example, if the ambient light is bright, the processor may reduce the ambient light transmittance, thereby reducing the amount of ambient light entering the user's eyes and thus increasing the brightness ratio of the virtual and real images entering the eyes; if the ambient light is dim, the processor may increase the ambient light transmittance, thereby increasing the amount of ambient light entering the user's eyes and thus decreasing the brightness ratio of the virtual and real images entering the eyes.

[0112] After adjusting the ambient light transmittance, the processor calculates the ratio of the virtual and real image brightness at eye level based on the adjusted ambient light transmittance and compares it with the preset brightness contrast. If the two are equal or close (within the allowable error range), the brightness adjustment is complete; if they are not equal, the virtual image brightness needs to be further adjusted.

[0113] Virtual image brightness refers to the brightness of the virtual image generated by the display module of a mixed reality device at the imaging location. By adjusting the virtual image brightness, the brightness ratio of the virtual and real images entering the eye can be directly affected.

[0114] For example, if the ratio of virtual to real image brightness entering the eye is still lower than the preset brightness contrast ratio after adjusting the ambient light transmittance, the processor may increase the virtual image brightness, thereby increasing the ratio of virtual to real image brightness entering the eye; if the ratio of virtual to real image brightness entering the eye is higher than the preset brightness contrast ratio, the processor may decrease the virtual image brightness, thereby decreasing the ratio of virtual to real image brightness entering the eye.

[0115] This embodiment, through the aforementioned adjustment process, enables the mixed reality device to adjust the ambient light transmittance and virtual image brightness based on the ambient light brightness and a preset brightness contrast ratio, making the ratio of the virtual and real image brightness at eye level close to the preset brightness contrast ratio, thus providing the best visual experience. Furthermore, the adjustment sequence of first adjusting the ambient light transmittance and then the virtual image brightness reduces device power consumption and ensures device performance.

[0116] Based on the above embodiments, this embodiment describes in detail how a mixed reality device can adjust brightness parameters while taking into account the limitations of the adjustment range. Figure 5The flowchart of the specific brightness parameter adjustment logic provided in this embodiment is as follows: Figure 5 As shown, the method includes:

[0117] Step 501: Determine the target transmittance value based on ambient light intensity, virtual image intensity, and preset brightness contrast.

[0118] In this step, the processor calculates the ideal ambient light transmittance, or target transmittance value, based on the ambient light brightness, virtual image brightness, and preset brightness contrast ratio. The target transmittance value refers to the ideal ambient light transmittance that ensures the ratio of the brightness of the virtual and real images entering the eye is equal to the preset brightness contrast ratio, without considering hardware limitations.

[0119] The target transmittance value can be calculated using the following formula:

[0120] Transmittance target value = Virtual image brightness / (Preset brightness contrast × Ambient light brightness);

[0121] Among them, virtual image brightness is the current virtual image brightness, preset brightness contrast is the brightness contrast set by the user or preset by the device, and ambient light brightness is the ambient light brightness measured by the ambient light sensor or camera module.

[0122] For example, if the current virtual image brightness is 100 nits, the preset brightness contrast ratio is 2, and the ambient light brightness is 500 nits, then the target transmittance value is 100 / (2×500)=0.1, which is 10%.

[0123] Step 502: If the transmittance target value is within the first adjustment range, adjust the ambient light transmittance to the transmittance target value; otherwise, adjust the ambient light transmittance to the first target endpoint value.

[0124] After calculating the target transmittance value, the processor checks whether the value is within the adjustment range of ambient light transmittance. The adjustment range of ambient light transmittance is determined by the hardware characteristics of the device, and usually has a minimum value and a maximum value, which constitute the first adjustment range.

[0125] If the target transmittance value is within the first adjustment range, the processor will directly adjust the ambient light transmittance to the target transmittance value. For example, if the first adjustment range is 5% to 80%, and the target transmittance value is 10%, the processor will adjust the ambient light transmittance to 10%.

[0126] If the target transmittance value is not within the first adjustment range, the processor will adjust the ambient light transmittance to the first target endpoint value. The first target endpoint value is the endpoint value with the smallest difference from the target transmittance value among the two endpoint values ​​of the first adjustment range.

[0127] For example, if the first adjustment range is 5% to 80%, and the transmittance target value is 2%, then the first target endpoint value is 5%, and the processor will adjust the ambient light transmittance to 5%; if the transmittance target value is 90%, then the first target endpoint value is 80%, and the processor will adjust the ambient light transmittance to 80%.

[0128] Step 503: If the ratio of the virtual and real image brightness at eye level obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast ratio, then determine the target value of the virtual image brightness based on the ambient light brightness, the adjusted ambient light transmittance, and the preset brightness contrast ratio.

[0129] After adjusting the ambient light transmittance, the processor calculates the ratio of the virtual to real image brightness at eye level based on the adjusted ambient light transmittance and compares it with a preset brightness contrast ratio. If the two are not equal, the virtual image brightness needs to be further adjusted.

[0130] The processor calculates the ideal virtual image brightness, or target value, based on the ambient light intensity, the adjusted ambient light transmittance, and the preset brightness contrast. The target value refers to the ideal virtual image brightness that, under the current ambient light transmittance, makes the ratio of the virtual image brightness to the brightness of the real image entering the eye equal to the preset brightness contrast.

[0131] The target value for virtual image brightness can be calculated using the following formula:

[0132] Virtual image brightness target value = preset brightness contrast ratio × ambient light brightness × adjusted ambient light transmittance;

[0133] For example, if the preset brightness contrast ratio is 2, the ambient light brightness is 500 nits, and the adjusted ambient light transmittance is 5%, then the target value for the virtual image brightness is 2 × 500 × 5% = 50 nits.

[0134] Step 504: If the target value of the virtual image brightness is within the second adjustment range, adjust the virtual image brightness to the target value; otherwise, adjust the virtual image brightness to the second target endpoint value.

[0135] After calculating the target value for virtual image brightness, the processor checks whether the value is within the adjustment range of virtual image brightness. The adjustment range of virtual image brightness is determined by the characteristics of the device's display module, and typically has a minimum value and a maximum value, constituting a second adjustment range.

[0136] If the target virtual image brightness value is within the second adjustment range, the processor will directly adjust the virtual image brightness to the target value. For example, if the second adjustment range is 10 nits to 500 nits, and the target virtual image brightness value is 50 nits, the processor will adjust the virtual image brightness to 50 nits.

[0137] If the target virtual image brightness value is not within the second adjustment range, the processor will adjust the virtual image brightness to the second target endpoint value. The second target endpoint value is the endpoint value with the smallest difference from the target virtual image brightness value among the two endpoint values ​​of the second adjustment range.

[0138] For example, if the second adjustment range is 10 nit to 500 nit, and the target value of the virtual image brightness is 5 nit, then the second target endpoint value is 10 nit, and the processor will adjust the virtual image brightness to 10 nit; if the target value of the virtual image brightness is 600 nit, then the second target endpoint value is 500 nit, and the processor will adjust the virtual image brightness to 500 nit.

[0139] It should be noted that a step-by-step adjustment method can be used when adjusting ambient light transmittance and virtual image brightness to avoid visual discomfort caused by abrupt changes in parameters.

[0140] Through the above steps, this embodiment enables the mixed reality device to adjust the ambient light transmittance and virtual image brightness as much as possible while taking into account hardware limitations, so that the brightness ratio of the virtual and real images entering the eye is close to the preset brightness contrast ratio, providing the best visual experience.

[0141] Based on the above embodiments, this embodiment describes in detail how the mixed reality device adjusts the brightness parameters when the target adjustment mode is the feature automatic tracking mode. Figure 6 This is a flowchart of the automatic feature tracking mode method provided in this embodiment, as follows: Figure 6 As shown, the method includes:

[0142] Step 601: Obtain and display the first environmental image output by the camera module of the mixed reality device.

[0143] In this step, the processor of the mixed reality device acquires the first environmental image output by the camera module through an internal communication interface. The first environmental image is an image of the current environment scene captured by the camera module, containing information about various objects and lighting in the environment.

[0144] The processor will present the initial environment image to the user, either directly on the mixed reality device's display or through a virtual window or interface within the user's field of vision. The display can be full-screen or shown in a specific area of ​​the user's field of vision, such as a corner or a floating window.

[0145] Step 602: In response to a user-triggered feature selection operation, determine at least one feature and the weight of each feature from the first environment image.

[0146] After displaying the initial environmental image, users can select features in the image through interactive methods such as clicking, selecting boxes, or voice commands. Features can be any object or area in the environment; in a surgical setting, features can be reference targets, guides, instruments, etc.

[0147] When a user triggers a feature selection operation, the processor responds by determining the selected feature from the first environment image. Feature determination can be achieved through methods such as image segmentation, object detection, or user-specified regions.

[0148] For example, if a user clicks on a location in an image, the processor can determine a fixed-size area centered on that location as a feature; if the user selects a region in the image, the processor can directly use that region as a feature; if the user selects a feature using a voice command such as "select a reference target," the processor can use an image recognition algorithm to identify the reference target region as a feature.

[0149] In addition to identifying features, the processor assigns a weight to each feature. The weight indicates the importance of the feature in brightness adjustment; the higher the weight, the greater the impact of the feature's brightness on the final ambient light brightness.

[0150] Weight allocation can be based on various factors, such as assigning weights according to the user's input operation on the weight of each feature; or automatically assigning weights based on the size, position, or order in which the user selects the features. For example, the processor can assign weights based on the area of ​​the feature, with larger features receiving higher weights; or based on the position of the feature in the image, with features located at the center of the image receiving higher weights; or based on the order in which the user selects the features, with earlier selected features receiving higher weights.

[0151] Step 603: Obtain the second environmental image output by the camera module.

[0152] After determining the features and weights, the processor continues to acquire the second environmental image output by the camera module. The second environmental image is an image of the environmental scene captured by the camera module after the features are determined. It may be the same as the first environmental image, or it may differ due to the passage of time or changes in the user's position.

[0153] The second environmental image is acquired in a similar way to the first environmental image, by obtaining image data from the camera module through an internal communication interface. The difference is that the second environmental image does not need to be displayed to the user; instead, it is used for subsequent brightness analysis.

[0154] Step 604: Determine the brightness and weight of each feature based on the second environmental image.

[0155] After acquiring the second environmental image, the processor analyzes the brightness of each feature in the image. Brightness can be obtained by calculating the brightness value of pixels within the feature region, such as the average brightness value, median brightness value, or weighted brightness value.

[0156] For example, the processor can convert the second environment image into a grayscale image and then calculate the average brightness value of all pixels within each feature region as the brightness of that feature. Alternatively, the processor can perform a weighted average of the pixel brightness within the feature region, giving higher weights to the central region and lower weights to the edge region to better reflect the brightness of the feature.

[0157] Simultaneously, the processor will confirm or update the weights of each feature. If the environment or a feature changes, the processor may need to adjust the weights to reflect the importance of the feature in the current environment. For example, if a feature becomes blurry or partially occluded in a second environmental image, the processor may decrease its weight; if a feature becomes clearer or occupies a larger area, the processor may increase its weight.

[0158] Step 605: Determine the ambient light brightness based on the brightness and weight of each feature.

[0159] After determining the brightness and weight of each feature, the processor calculates the ambient light brightness based on this information. The ambient light brightness can be calculated using a weighted average method, which involves multiplying the brightness of each feature by its weight and then summing the results.

[0160] Ambient light intensity = Σ(intensity of feature i × weight of feature i);

[0161] Where Σ represents summation, and i represents the index of the feature.

[0162] For example, if there are three features with brightness values ​​of 100 nit, 200 nit, and 300 nit, and weights of 0.5, 0.3, and 0.2, respectively, then the ambient light brightness is 100×0.5+200×0.3+300×0.2=50+60+60=170 nit.

[0163] Step 606: Adjust the brightness parameters of the mixed reality device according to the ambient light intensity.

[0164] After determining the ambient light brightness, the processor can adjust the brightness parameters of the mixed reality device according to a preset brightness adjustment strategy or preset rules, which will not be described in detail in this embodiment.

[0165] The automatic feature tracking mode can analyze the environmental images output by the camera module in real time and dynamically adjust the brightness parameters. For example, when the camera module outputs each frame of the environmental image, the processor will recalculate the ambient light brightness and adjust the brightness parameters accordingly, providing more real-time and precise brightness adjustment.

[0166] Through the above steps, this embodiment enables the mixed reality device to analyze the brightness of the features selected by the user and adjust the brightness parameters accordingly, providing more personalized and precise brightness adjustment.

[0167] Based on the above embodiments, this embodiment describes in detail the method for determining the weight of features in the automatic feature tracking mode.

[0168] In automatic feature tracking mode, the weights of features significantly impact the accuracy of brightness adjustment. This embodiment provides a weight allocation method based on selection order, enabling the mixed reality device to rationally allocate feature weights according to the user's selection preferences.

[0169] Specifically, the weights of each feature decrease sequentially according to the order in which they are selected, and the sum of their weights equals 1. This weighting method is based on the assumption that the feature selected first by the user is usually the most important and best represents the ambient light conditions that the user is interested in.

[0170] For example, if a user selects three features A, B, and C in sequence, their weights can be allocated as follows: the weight of feature A = 0.5; the weight of feature B = 0.3; and the weight of feature C = 0.2. Thus, the sum of the weights of the three features is 0.5 + 0.3 + 0.2 = 1.

[0171] If the user selects more features, the weights can be distributed in a similar decreasing manner. For example, if the user selects five features A, B, C, D, and E, their weights could be: feature A = 0.4; feature B = 0.3; feature C = 0.15; feature D = 0.1; feature E = 0.05; thus, the sum of the weights of the five features is still 0.4 + 0.3 + 0.15 + 0.1 + 0.05 = 1.

[0172] This embodiment uses this weighting method to enable the mixed reality device to better reflect the user's selection preferences, allocating more weight to the features selected by the user first, thereby providing more personalized brightness adjustment.

[0173] Based on the above embodiments, this embodiment describes in detail how the mixed reality device adjusts the brightness parameters when the target adjustment mode is the focus point automatic tracking mode. Figure 7 This is a flowchart of the automatic focus tracking mode method provided in this embodiment, as follows: Figure 7 As shown, the method includes:

[0174] Step 701: Acquire the user's binocular focus points and the third environment image output by the camera module of the mixed reality device.

[0175] In this step, the mixed reality device uses eye-tracking technology to obtain the user's eye focus point. Eye-tracking technology uses a built-in eye tracker to identify the user's pupil position and gaze direction, determining the location of the user's focus point in three-dimensional space.

[0176] Meanwhile, the processor of the mixed reality device acquires a third-environment image output by the camera module through an internal communication interface. The third-environment image is an image of the current environment scene captured by the camera module, containing information about various objects and lighting in the environment.

[0177] Step 702: Based on the binocular focal points, determine the focal range and the brightness of the focal range from the third ambient image, and use the brightness of the focal range as the ambient light brightness.

[0178] After acquiring the user's binocular focus points and the third-environment image, the processor maps the focus points in 3D space onto the third-environment image to determine the position of the focus points in the image. Then, a focus range is determined with the focus points as the center.

[0179] The focal range can be a fixed-size area, such as a circular or rectangular area centered on the focal point; or it can be an area whose size is dynamically adjusted according to the focal distance, such as the larger the focal range, the farther the focal distance.

[0180] For example, the processor can determine a circular area with a radius of 10% of the image width as the focus range, centered on the focus point; or, the processor can determine a circular area with a radius of k×d as the focus range based on the focus distance d, where k is a scaling factor.

[0181] After determining the focus area, the processor calculates the brightness within that area. Brightness can be obtained by calculating the brightness values ​​of the pixels within the focus area, such as average brightness value, median brightness value, or weighted brightness value.

[0182] For example, the processor can convert the third-party ambient image into a grayscale image and then calculate the average brightness value of all pixels within the focal range as the brightness of that range. Alternatively, the processor can perform a weighted average of the pixel brightness within the focal range, giving higher weights to the central region and lower weights to the edge region, to better reflect the brightness of the focal range.

[0183] The processor will use the brightness of the focused area as the ambient light brightness for subsequent brightness adjustments.

[0184] Step 703: Adjust the brightness parameters of the mixed reality device according to the ambient light intensity.

[0185] After determining the ambient light brightness, the processor can adjust the brightness parameters of the mixed reality device according to a preset brightness adjustment strategy or preset rules, which will not be described in detail in this embodiment.

[0186] The auto-focus tracking mode can track the user's gaze in real time and dynamically adjust the brightness parameters. When the user's gaze moves to different areas, the processor recalculates the brightness of the focus range and adjusts the brightness parameters accordingly, providing more real-time and precise brightness adjustment.

[0187] Through the above steps, this embodiment enables the mixed reality device to analyze the brightness of the focal range based on the user's eye focus points and adjust the brightness parameters accordingly, providing dynamic brightness adjustment that follows the user's gaze and adapts to scenarios where the user's gaze changes frequently.

[0188] Based on the above embodiments, this embodiment further illustrates the brightness adjustment method for mixed reality devices in the context of a surgical scenario. In this embodiment, the method includes:

[0189] Step 1: Device initialization.

[0190] After the user starts the device, the device automatically performs the following operations: loads the preset virtual image brightness baseline value (e.g., 200 nits) and the default ambient light transmittance value (e.g., 50%). Activates the sensor network (ambient light sensor, camera module, eye tracker) and completes calibration. A brightness adjustment mode selection menu pops up on the AR display interface, containing the following options: Mode 1: Manual adjustment; Mode 2: Ambient light sensing; Mode 3: Automatic feature tracking; Mode 4: Automatic focus tracking.

[0191] Step 2: Mode selection and parameter pre-configuration.

[0192] Users select the target adjustment mode via voice commands (such as "switch to feature tracking mode") or gestures (such as pinch gestures to select menu items) and configure the virtual image brightness and / or preset brightness contrast: set the virtual image brightness (such as 100 nits) and the preset brightness contrast (such as 2).

[0193] Step 3: Adjustment mode execution.

[0194] Execute the differentiated process based on the selected pattern:

[0195] If manual adjustment mode is selected, the specific process is as follows: The user inputs adjustment commands via physical buttons (brightness + / - buttons), gestures (sliding the virtual adjustment bar), or voice (e.g., "increase brightness by 20%"). The processor parses the commands and generates control signals: it prioritizes adjusting the ambient light transmittance (e.g., in 1% increments); if the ambient light transmittance has reached its limit (e.g., 10% or 90%), it adjusts the virtual image brightness accordingly (e.g., in 10 nit increments), and overlays adjustment effect prompts in real time (e.g., a semi-transparent progress bar).

[0196] If the ambient light sensing mode is selected, the specific process can be as follows: the camera module acquires ambient images at a fixed frame rate, and the processor performs the following analysis: for camera modules using a fixed exposure mode, the ambient light brightness is calculated based on the average pixel value of the entire image or the region of interest; for camera modules using an adaptive exposure mode, the camera exposure time parameters are extracted, and the corresponding ambient light brightness is determined according to a preset brightness mapping table.

[0197] If the automatic feature tracking mode is selected, the user selects features (such as surgical instruments or anatomical landmarks) in the environment image using gestures and sets weights (which are assigned by default according to the selection order). The system calls a model such as YOLOv5 to extract the bounding boxes of the features in real time, tracks their position changes, calculates the weighted average of pixel brightness within each feature region, and then dynamically adjusts the ambient light transmittance and virtual image brightness.

[0198] If the automatic focus tracking mode is selected, the eye tracker outputs the focus coordinates at a fixed frequency. The coordinates are then transformed to the camera coordinate system using a calibration matrix. Based on the camera intrinsic parameter matrix, the focus coordinates are projected onto the image plane to obtain pixel coordinates. The average pixel value of the annular region centered on these pixel coordinates is extracted, local brightness is calculated, and the ambient light transmittance and virtual image brightness are adjusted according to the local brightness.

[0199] It should be noted that in adaptive adjustment mode, a dynamic priority control logic is used, meaning that ambient light transmittance adjustment takes precedence over virtual image brightness adjustment, and virtual image brightness adjustment is only initiated after the ambient light transmittance reaches a threshold. During adaptive adjustment, users can reset the brightness parameters to safe parameters at any time using an emergency command (such as long-pressing a physical button).

[0200] Based on the above embodiments, this embodiment further illustrates the brightness adjustment method for mixed reality devices in the context of a surgical scenario. In this embodiment, the method includes:

[0201] Step 1: Device initialization.

[0202] After the user starts the device, the device automatically performs the following operations: initialize device parameters, such as the initial value of virtual image brightness (T_init), the target value of brightness contrast (B_tar), and the initial value of ambient light transmittance (K_init). The virtual image brightness (T) has a maximum extreme value (T_max) and a minimum extreme value (T_min), and the ambient light transmittance (K) has a maximum extreme value (K_max) and a minimum extreme value (K_min).

[0203] Step 2: User mode selection.

[0204] The user selects whether to enable the adaptive adjustment mode. If the user selects to disable the adaptive adjustment mode, the process enters the manual adjustment mode, where the user manually sets the ambient light transmittance (K) and the virtual image brightness (T); if the user enables the adaptive adjustment mode, the user selects whether to enable the camera tracking mode.

[0205] If the user selects to disable the camera tracking mode, the device uses the output data of the ambient light sensor as the adjustment basis, monitors the difference between the current brightness contrast (B) and the target brightness contrast value (B_tar) in real time, and prioritizes adjusting the ambient light transmittance (K). If the extreme value is not reached (K_min<K<K_max), K is continuously adjusted until B=B_tar; if B≠B_tar when K reaches the extreme value, the virtual image brightness (T) is adjusted until B=B_tar is satisfied or T reaches the extreme value.

[0206] If the user selects to enable the camera tracking mode, the device automatically tracks the highlight area in the field of view by default, and the user can also mark a specific feature to be tracked. If the user enables the camera adaptive exposure, the ambient light brightness can be evaluated according to the current exposure parameters; if the adaptive exposure is disabled, the ambient light brightness can be evaluated according to the image pixel brightness of the tracking area. The logic of performing brightness adjustment based on the output data (exposure parameters or pixel brightness) of the camera module is similar to the logic of performing brightness adjustment based on the output data of the ambient light sensor, and will not be repeated here.

[0207] It should be understood that although Figure 1-7 the respective steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict sequence restriction on the execution of these steps, and these steps may be executed in other orders. Moreover, Figure 1-7 at least part of the steps may include a plurality of steps or a plurality of stages, these steps or stages are not necessarily completed at the same moment, but may be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with at least a part of other steps or steps or stages in other steps.

[0208] This embodiment provides a mixed reality device. Figure 8 This is a functional component architecture diagram of the mixed reality device provided in this embodiment, such as... Figure 8 As shown, the mixed reality device 800 includes a processor 801, a display module 802, an optical lens 803, an ambient light sensor 804, and a camera module 805.

[0209] The processor 801 is the core control unit of the mixed reality device 800, responsible for processing various data and performing various functions. The processor 801 can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, etc.

[0210] Display module 802 is used to display virtual images. It can employ technologies such as optical waveguides, holographic optical elements (HOE), or freeform surface optics to project the virtual image into the user's field of vision, blending it with the real environment to create a mixed reality experience. The output brightness of display module 802 can be controlled by processor 801 to adjust the brightness of the virtual image.

[0211] Optical lens 803 is used to protect the user's eyes. Optical lens 803 can use technologies such as electrochromic materials, liquid crystal dimming materials or mechanical apertures to make its transmittance adjustable, so as to control the amount of ambient light entering the user's eyes.

[0212] An ambient light sensor 804 is mounted on the outer surface of the mixed reality device 800 to detect the ambient light level. The ambient light sensor 804 transmits the detection results to the processor 801 in the form of a digital signal, and the processor 801 adjusts the brightness parameters based on this data.

[0213] A camera module 805 is mounted on the front of the mixed reality device 800 to capture images within the user's field of view. The camera module 805 can be an RGB camera, a depth camera, or a combination of both. The image data captured by the camera module 805 is transmitted to a processor 801, which extracts lighting information using image processing algorithms and adjusts brightness parameters accordingly.

[0214] The mixed reality device 800 may also include other components such as eye trackers, inertial measurement units, speakers, microphones, batteries, etc., to provide a complete mixed reality experience.

[0215] In the mixed reality device 800, the processor 801 is configured to perform the following operations: in response to a user-triggered mode selection operation, determine a target adjustment mode from a set of preset brightness adjustment modes; wherein the set of brightness adjustment modes includes a manual adjustment mode and an adaptive adjustment mode; and adjust the brightness parameters of the mixed reality device 800 based on the target adjustment mode.

[0216] The specific adjustment process is similar to that described in the above embodiments. The processor 801 will use different strategies to adjust the brightness parameters of the mixed reality device 800 according to different adjustment modes in order to provide the best visual experience.

[0217] For example, in manual adjustment mode, processor 801 responds to the user's brightness adjustment command and directly adjusts the brightness parameters of the mixed reality device 800; in ambient light sensing mode, processor 801 automatically adjusts the brightness parameters based on the ambient light brightness sensed by the ambient light sensor; in camera tracking mode, processor 801 analyzes the brightness of a specific area based on the environmental image captured by camera module 805 and adjusts the brightness parameters accordingly.

[0218] The mixed reality device provided in this embodiment includes a manual adjustment mode and an adaptive adjustment mode. The manual adjustment mode allows users to directly adjust the device's brightness parameters, while the adaptive adjustment mode automatically optimizes the brightness parameters by sensing changes in ambient light in real time, requiring no manual intervention. After the user selects a target adjustment mode, the device's processor will execute the corresponding brightness adjustment operation according to the mode type. This design achieves a dual advantage: on the one hand, it retains the flexibility of manual adjustment to meet personalized needs; on the other hand, the adaptive mode, through intelligent brightness adjustment, eliminates the risk of distraction for doctors during surgery while ensuring that the virtual image remains clearly visible under various lighting conditions. This dynamic adjustment mechanism significantly improves the focus, smoothness, and accuracy of surgical operations, providing a more reliable solution for the application of mixed reality technology in medical scenarios.

[0219] This embodiment provides a mixed reality surgical navigation system. Figure 9 The functional component architecture diagram of the mixed reality surgical navigation system provided in this embodiment is as follows: Figure 9 As shown, the mixed reality surgical navigation system 900 includes the mixed reality device 800 described in the above embodiments. This system applies mixed reality technology to the field of surgical navigation, providing doctors with real-time surgical guidance and assistance.

[0220] In a surgical environment, lighting conditions can change frequently, such as switching surgical lights on and off and adjusting ambient lighting. Simultaneously, doctors need to monitor different areas, such as the patient's surgical site, surgical instruments, and monitoring equipment. These factors all pose challenges to the brightness adjustment of the Mixed Reality 800 device.

[0221] The mixed reality device 800 in the mixed reality surgical navigation system 900 can provide multiple brightness adjustment modes to meet the special needs of the surgical environment. For example:

[0222] In manual adjustment mode, doctors can manually adjust the brightness parameters of the mixed reality device 800 according to their personal preferences and current surgical needs. For example, they can increase the brightness of the virtual image when they need to focus on virtual navigation information, and increase the ambient light transmittance when they need to focus on the patient's actual condition.

[0223] In ambient light sensing mode, the Mixed Reality Device 800 can automatically adjust brightness parameters according to the overall lighting conditions of the operating room, adapting to the switching on and off of surgical lights and changes in indoor lighting, providing doctors with a comfortable visual experience.

[0224] In camera tracking mode, the mixed reality device 800 can analyze the brightness of specific areas of focus for the doctor, such as the surgical site, surgical instruments, or monitoring equipment, and adjust the brightness parameters accordingly, providing more precise brightness control. Especially in automatic focus tracking mode, the mixed reality device 800 can track the doctor's gaze in real time and dynamically adjust the brightness parameters to accommodate situations where the doctor needs to frequently switch their focus during surgery.

[0225] The mixed reality surgical navigation system provided in this embodiment includes a manual adjustment mode and an adaptive adjustment mode. The manual adjustment mode allows users to directly adjust the device's brightness parameters, while the adaptive adjustment mode automatically optimizes the brightness parameters by sensing changes in ambient light in real time, requiring no manual intervention. After the user selects a target adjustment mode, the device's processor will execute the corresponding brightness adjustment operation according to the mode type. This design achieves a dual advantage: on the one hand, it retains the flexibility of manual adjustment to meet personalized needs; on the other hand, the adaptive mode, through intelligent brightness adjustment, eliminates the risk of distraction for doctors during surgery while ensuring that the virtual image remains clearly visible under various lighting conditions. This dynamic adjustment mechanism significantly improves the focus, smoothness, and accuracy of surgical operations, providing a more reliable solution for the application of mixed reality technology in medical scenarios.

[0226] This embodiment provides a brightness adjustment device for a mixed reality device. Figure 10 A block diagram of the brightness adjustment device provided in this embodiment, as shown below. Figure 10 As shown, the brightness adjustment device 1000 includes: a mode determination module 1001 and a brightness adjustment module 1002. Wherein:

[0227] The mode determination module 1001 is used to determine a target adjustment mode from a variety of preset brightness adjustment modes in response to a user-triggered mode selection operation; wherein the various brightness adjustment modes include a manual adjustment mode and an adaptive adjustment mode. The brightness adjustment module 1002 is used to adjust the brightness parameters of the mixed reality device based on the target adjustment mode.

[0228] The mode determination module 1001 and the brightness adjustment module 1002 can be functional modules implemented by the processor in the mixed reality device by executing specific program code, or they can be independent hardware modules or a combination of both.

[0229] The mode determination module 1001 is responsible for receiving the user's mode selection operation and determining the target adjustment mode selected by the user from a variety of preset brightness adjustment modes. The mode selection operation can be triggered by physical buttons, touch screen, voice commands, gesture recognition, or eye tracking. The preset brightness adjustment modes include manual adjustment mode and adaptive adjustment mode, and the adaptive adjustment mode can further include ambient light sensing mode and camera tracking mode, etc.

[0230] The brightness adjustment module 1002 is responsible for adjusting the brightness parameters of the mixed reality device according to the target adjustment mode. Brightness parameters may include ambient light transmittance and virtual image brightness, etc. The brightness adjustment module 1002 can adopt different adjustment strategies according to different adjustment modes; for example, in manual adjustment mode, it responds to the user's direct adjustment operation, and in adaptive adjustment mode, it automatically adjusts according to ambient light conditions.

[0231] In one embodiment, the target adjustment mode is an ambient light sensing mode; the brightness adjustment module 1002 includes: a first data acquisition submodule, used to acquire the output data of the ambient light sensor of the mixed reality device; a first brightness determination submodule, used to determine the ambient light brightness within the sensing range of the ambient light sensor based on the output data; and a first adjustment submodule, used to adjust the brightness parameters of the mixed reality device based on the ambient light brightness.

[0232] In one embodiment, the target adjustment mode is a camera tracking mode; the brightness adjustment module 1002 includes: a second data acquisition submodule for acquiring output data from the camera module of the mixed reality device; a second brightness determination submodule for determining the ambient light brightness within the target imaging range of the camera module based on the output data; and a second adjustment submodule for adjusting the brightness parameters of the mixed reality device based on the ambient light brightness.

[0233] In one embodiment, the target imaging range is: the global imaging range of the camera module; or, within the global imaging range, at least one range where the brightness meets a preset brightness condition; or, at least one range determined by the user within the global imaging range; or, the user's binocular focusing range.

[0234] In one embodiment, the brightness parameters include ambient light transmittance and virtual image brightness; the first adjustment submodule or the second adjustment submodule includes: a preset contrast acquisition unit, used to acquire a preset brightness contrast of the mixed reality device; and a parameter adjustment unit, used to adjust the brightness parameters of the mixed reality device according to the ambient light brightness and the preset brightness contrast, based on a preset rule; wherein the preset rule is: first adjust the ambient light transmittance, and if the ratio of the virtual and real image brightness at eye level obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast, then adjust the virtual image brightness.

[0235] In one embodiment, the ambient light transmittance has a first adjustment range, the first adjustment range having two first endpoint values; the virtual image brightness has a second adjustment range, the second adjustment range having two second endpoint values; the parameter adjustment unit includes: a transmittance target value determination subunit, configured to determine a transmittance target value based on the ambient light brightness and a preset brightness contrast ratio; and a transmittance adjustment subunit, configured to adjust the ambient light transmittance to the transmittance target value if the transmittance target value is within the first adjustment range; otherwise, adjust the ambient light transmittance to the first target endpoint value; wherein the first target endpoint value is the transmittance value between the two first endpoint values. The first endpoint value with the smallest difference from the target value; the virtual image brightness target value determination subunit, used to determine the virtual image brightness target value based on the ambient light brightness, the adjusted ambient light transmittance and the preset brightness contrast if the virtual and real image brightness ratio obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast; the virtual image brightness adjustment subunit, used to adjust the virtual image brightness to the virtual image brightness target value if the virtual image brightness target value is within the second adjustment range; otherwise, adjust the virtual image brightness to the second target endpoint value; wherein, the second target endpoint value is the second endpoint value with the smallest difference from the virtual image brightness target value among the two second endpoint values.

[0236] In one embodiment, the camera tracking mode includes an automatic feature tracking mode, and the target adjustment mode is the automatic feature tracking mode; the brightness adjustment module 1002 includes: an image display submodule, used to acquire and display a first environmental image output by the camera module of the mixed reality device; a feature determination submodule, used to determine at least one feature and the weight of each feature from the first environmental image in response to a feature selection operation triggered by a user; an image acquisition submodule, used to acquire a second environmental image output by the camera module; a feature brightness determination submodule, used to determine the brightness and weight of each feature based on the second environmental image; a first ambient light brightness determination submodule, used to determine the ambient light brightness based on the brightness and weight of each feature; and a first brightness parameter adjustment submodule, used to adjust the brightness parameter of the mixed reality device based on the ambient light brightness.

[0237] In one embodiment, the weights of each feature decrease sequentially according to the selection order, and the sum of the weights of each feature equals 1.

[0238] In one embodiment, the camera tracking mode includes an automatic focus tracking mode, and the target adjustment mode is an automatic focus tracking mode; the brightness adjustment module 1002 includes: a focus acquisition submodule, used to acquire the user's binocular focus points and a third environmental image output by the camera module of the mixed reality device; a second ambient light brightness determination submodule, used to determine the focus range and the brightness of the focus range from the third environmental image based on the binocular focus points, and use the brightness of the focus range as the ambient light brightness; and a second brightness parameter adjustment submodule, used to adjust the brightness parameters of the mixed reality device based on the ambient light brightness.

[0239] It should be noted that the specific limitations regarding the brightness adjustment mechanism of the mixed reality device can be found in the limitations on the brightness adjustment method described above, and will not be repeated here. Each module in the aforementioned brightness adjustment mechanism can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0240] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0241] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0242] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0243] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0244] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for adjusting the brightness of a mixed reality device, characterized in that, The method includes: In response to a user-triggered mode selection operation, a target adjustment mode is determined from a variety of preset brightness adjustment modes; wherein, the variety of brightness adjustment modes includes a manual adjustment mode and an adaptive adjustment mode; The brightness parameters of the mixed reality device are adjusted based on the target adjustment mode.

2. The brightness adjustment method for a mixed reality device as described in claim 1, characterized in that, The adaptive adjustment modes include ambient light sensing mode and camera tracking mode.

3. The brightness adjustment method for a mixed reality device as described in claim 2, characterized in that, The target adjustment mode is the ambient light sensing mode; adjusting the brightness parameters of the mixed reality device based on the target adjustment mode includes: Acquire the output data of the ambient light sensor of the mixed reality device; Based on the output data, the ambient light brightness within the sensing range of the ambient light sensor is determined; The brightness parameters of the mixed reality device are adjusted according to the ambient light intensity.

4. The brightness adjustment method for a mixed reality device as described in claim 2, characterized in that, The target adjustment mode is the camera tracking mode; adjusting the brightness parameters of the mixed reality device based on the target adjustment mode includes: Obtain the output data of the camera module of the mixed reality device; Based on the output data, determine the ambient light intensity within the target imaging range of the camera module; The brightness parameters of the mixed reality device are adjusted according to the ambient light intensity.

5. The brightness adjustment method for a mixed reality device as described in claim 4, characterized in that, The target imaging range is: The global imaging range of the camera module; or... Within the global imaging range, the brightness satisfies at least one range of preset brightness conditions; or, At least one range defined by the user within the global imaging range; or... The user's binocular focusing range.

6. The brightness adjustment method for a mixed reality device as described in claim 3 or 4, characterized in that, The brightness parameters include ambient light transmittance and virtual image brightness; adjusting the brightness parameters of the mixed reality device according to the ambient light brightness includes: Obtain the virtual image brightness and preset brightness contrast of the mixed reality device; The brightness parameters of the mixed reality device are adjusted according to the ambient light brightness, the virtual image brightness, and the preset brightness contrast ratio, using a preset rule. The preset rule is as follows: first, the ambient light transmittance is adjusted; if the ratio of the virtual and real image brightness at eye level obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast ratio, then the virtual image brightness is adjusted.

7. The brightness adjustment method for a mixed reality device as described in claim 6, characterized in that, The ambient light transmittance has a first adjustment range, the first adjustment range has two first endpoint values, and the virtual image brightness has a second adjustment range, the second adjustment range has two second endpoint values. The step of adjusting the brightness parameters of the mixed reality device according to the ambient light brightness, the virtual image brightness, and the preset brightness contrast according to preset rules includes: The target transmittance value is determined based on the ambient light intensity, the virtual image intensity, and the preset brightness contrast. If the transmittance target value is within the first adjustment range, then the ambient light transmittance is adjusted to the transmittance target value; otherwise, the ambient light transmittance is adjusted to the first target endpoint value; wherein, the first target endpoint value is the first endpoint value with the smallest difference from the transmittance target value among the two first endpoint values. If the ratio of the virtual and real image brightness at eye level obtained based on the adjusted ambient light transmittance is not equal to the preset brightness contrast ratio, then the target brightness value is determined based on the ambient light brightness, the adjusted ambient light transmittance, and the preset brightness contrast ratio. If the target brightness value is within the second adjustment range, the virtual image brightness is adjusted to the target brightness value; otherwise, the virtual image brightness is adjusted to the second target endpoint value; wherein, the second target endpoint value is the second endpoint value among the two second endpoint values ​​that has the smallest difference from the target brightness value.

8. The brightness adjustment method for a mixed reality device as described in claim 2, characterized in that, The camera tracking mode includes an automatic feature tracking mode, and the target adjustment mode is the automatic feature tracking mode; Adjusting the brightness parameters of the mixed reality device based on the target adjustment mode includes: Acquire and display the first environmental image output by the camera module of the mixed reality device; In response to a user-triggered feature selection operation, at least one feature and a weight for each feature are determined from the first environmental image; Acquire the second environmental image output by the camera module; Based on the second environmental image, determine the brightness and weight of each feature; The ambient light brightness is determined based on the brightness and weight of each of the aforementioned features; The brightness parameters of the mixed reality device are adjusted according to the ambient light intensity.

9. The brightness adjustment method for a mixed reality device as described in claim 8, characterized in that, According to the selection order of each feature, the weight of each feature decreases sequentially, and the sum of the weights of each feature is equal to 1.

10. The brightness adjustment method for a mixed reality device as described in claim 2, characterized in that, The camera tracking mode includes an automatic focus tracking mode, and the target adjustment mode is the automatic focus tracking mode; adjusting the brightness parameters of the mixed reality device based on the target adjustment mode includes: Acquire the user's binocular focus points and the third environmental image output by the camera module of the mixed reality device; Based on the binocular focal points, the focal range and the brightness of the focal range are determined from the third environmental image, and the brightness of the focal range is used as the ambient light brightness. The brightness parameters of the mixed reality device are adjusted according to the ambient light intensity.

11. A mixed reality device, characterized in that, It includes a processor, a display module, optical lenses, an ambient light sensor, and a camera module; the processor is used to perform: In response to a user-triggered mode selection operation, a target adjustment mode is determined from a variety of preset brightness adjustment modes; wherein, the variety of brightness adjustment modes includes a manual adjustment mode and an adaptive adjustment mode; The brightness parameters of the mixed reality device are adjusted based on the target adjustment mode.

12. A mixed reality surgical navigation system, characterized in that, Includes the mixed reality device as described in claim 11.

13. A brightness adjustment device for a mixed reality device, characterized in that, The device includes: The mode determination module is used to determine the target adjustment mode from a variety of preset brightness adjustment modes in response to a mode selection operation triggered by the user; wherein, the variety of brightness adjustment modes includes a manual adjustment mode and an adaptive adjustment mode; A brightness adjustment module is used to adjust the brightness parameters of the mixed reality device based on the target adjustment mode.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the brightness adjustment method for a mixed reality device as described in any one of claims 1 to 10.

Images