Mixed reality display method, mixed reality device and storage medium

Abstract

A mixed reality display method, mixed reality device and storage medium, the method comprising: after receiving a mixed display signal, tracking the user's eyeballs through an eye tracker to determine the user's gaze point on the display screen (101); determining the sub-display area that the user is gazing at based on the correspondence between the gaze point and the sub-display area in the display screen (102); determining at least one first image sensor based on the correspondence between the sub-display area and the image sensors in the image sensor array, and adjusting the at least one first image sensor to increase the resolution of an image generated by the at least one first image sensor; wherein the distance between the center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and the field of view angles of two adjacent image sensors in the image sensor array partially overlap (103); and superimposing and rendering an environmental image output by the image sensor array and a virtual image to obtain and display an MR image (104).

Description

Technical Field

[0001] The present disclosure relates to the field of mixed reality, and in particular to a mixed reality display method, a mixed reality device, and a storage medium. Background Art

[0002] The ultimate goal of mixed reality (MR) is to integrate the virtual world and the real world.

[0003] The most commonly used technology now is achieved through MR glasses. The optical components of MR glasses superimpose the information of the virtual world on the lighting information of the real world, and input them into the human eyes to achieve a mixed display.

[0004] In existing technologies, mainstream MR glasses mainly use holographic projection technology to achieve the superposition and combination of the virtual world and the real world. The disadvantages of this method are small field of view and insufficient depth of field. Summary of the Invention

[0005] The present disclosure provides a mixed reality display method, a mixed reality device, and a storage medium to solve the technical problems in the prior art of mixed reality display, such as a small field of view and insufficient depth of field.

[0006] In a first aspect, to address the above-mentioned technical problems, embodiments of the present disclosure provide a method for mixed reality display, which is applied to an MR device. The MR device includes a display screen, an eye tracker mounted on a display surface of the display screen, and an image sensor array mounted on a non-display surface of the display screen. The image sensors in the image sensor array correspond to the multiple sub-display areas of the display screen. The technical solution of this method is as follows:

[0007] After receiving the mixed display signal, tracking the user's eyes through the eye tracker to determine the user's gaze point on the display screen;

[0008] determining the sub-display area gazed by the user according to a correspondence between the gaze point and the sub-display area in the display screen;

[0009] determining at least one first image sensor based on a correspondence between the sub-display areas and image sensors in the image sensor array, and adjusting the at least one first image sensor to increase a resolution of an image generated by the at least one first image sensor; wherein a distance between a center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and field of view angles of two adjacent image sensors in the image sensor array partially overlap;

[0010] The environmental image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image.

[0011] In one possible implementation, adjusting the at least one first image sensor includes:

[0012] Determining whether a gaze time duration during which the gaze point stays in the sub-display area corresponding to the at least one first image sensor is greater than a preset threshold;

[0013] If the gaze duration is less than or equal to the preset threshold, increasing the resolution of the image generated by the at least one first image sensor to improve the resolution of the corresponding image;

[0014] If the gaze duration is greater than the preset threshold, the at least one first image sensor is kept in a working state, and the image output by the at least one first sensor is displayed in full screen as the environment image.

[0015] In one possible implementation, before superimposing and rendering the environment image output by the image sensor array with the virtual image, the method further includes:

[0016] If the total number of image sensors in the image sensor array that are in a working state is greater than 1, stitching a plurality of images output by the image sensor array into a stitched image;

[0017] The brightness and color of the stitched image are adjusted to generate the environment image.

[0018] In a possible implementation, stitching a plurality of images output by the image sensor array into a stitched image includes:

[0019] Selecting an image from the multiple images as a reference image, and stitching two adjacent images starting from the reference image until the stitching of the multiple images output by the image sensor array is completed;

[0020] Among them, the following method is used to stitch two adjacent images:

[0021] Acquire multiple matching points having the same image features from the two adjacent images;

[0022] Calculating the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of the plurality of matching points;

[0023] Calculating the homography matrix of each matching point according to the translation matrix, the rotation matrix and the intrinsic parameter matrix of each matching point;

[0024] Calculating each homography matrix using a minimum median robustness method, selecting matching points corresponding to the homography matrix that meets preset quality requirements to form a best subset, and calculating a final homography matrix based on the best subset so that the two adjacent images become images with the same perspective;

[0025] Aligning the coordinates of the pixels in the images with the same viewing angle, and averaging the pixels in the overlapping portion to obtain a sub-mosaic image of the two adjacent images;

[0026] The stitching process of the two adjacent images is repeated for all the sub-stitched images to obtain the stitched image.

[0027] In one possible implementation, adjusting the brightness and color of the stitched image to generate the environment image includes:

[0028] determining a blurred area in the stitched image according to a change in the optical flow field in the stitched image;

[0029] performing bilateral filtering on the blurred area to obtain a filtered image;

[0030] Performing illumination correction on the filtered image using a preset illumination model;

[0031] The overlapping areas of the two adjacent images corresponding to the blurred area are fused according to the weights to obtain the environment image.

[0032] In one possible implementation, the environmental image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image, including:

[0033] When the mixed display signal is a dynamic mixed display signal, determining a position of a preset object in the environment image;

[0034] Moving the virtual image of the virtual object in the current frame to the position, and adjusting the angle and size of the virtual image;

[0035] According to the detected lighting of the real environment, the lighting and shadow effects of the virtual image are adjusted to obtain the MR image, and the MR image is transmitted to the display screen for display.

[0036] In one possible implementation, determining a position of a preset object in the environment image includes:

[0037] extracting a plurality of feature points from the environment image;

[0038] Matching the plurality of feature points with feature points of the preset object;

[0039] The position of the successfully matched feature point in the environmental image is determined as the position of the preset object in the environmental image.

[0040] In one possible implementation, the environmental image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image, including:

[0041] When the mixed display signal is a static mixed display signal, a static virtual image is superimposed and rendered into the environmental image to obtain and display the MR image.

[0042] In a possible implementation, when a virtual display signal is received, the image sensor array is turned off and the image of the virtual world is displayed.

[0043] In a possible implementation, when a real scene display signal is received, the environment image is displayed.

[0044] In a second aspect, an embodiment of the present disclosure provides a mixed reality device, including:

[0045] A display screen, an eye tracker mounted on a display surface of the display screen, and an image sensor array mounted on a non-display surface of the display screen; wherein the image sensors in the image sensor array correspond to the plurality of sub-display areas included in the display screen;

[0046] The driving circuit is configured to, after receiving a mixed display signal, control the eye tracker to track the user's eyes to determine the user's gaze point on the display screen; determine at least one first image sensor corresponding to the gaze point based on a correspondence between the gaze point and a sub-display area in the display screen, and a correspondence between the sub-display area and image sensors in an image sensor array; and adjust the at least one first image sensor to increase the resolution of an image generated by the at least one first image sensor; wherein the distance between a center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and the field of view angles of two adjacent image sensors in the image sensor array partially overlap; and superimpose and render an environmental image output by the image sensor array and a virtual image to obtain and display an MR image.

[0047] In one possible implementation manner, the driving circuit is further configured to:

[0048] Determining whether a gaze time duration during which the gaze point stays in the sub-display area corresponding to the at least one first image sensor is greater than a preset threshold;

[0049] If the gaze duration is less than or equal to the preset threshold, increasing the resolution of the image generated by the at least one first image sensor to improve the resolution of the corresponding image;

[0050] If the gaze duration is greater than the preset threshold, the at least one first image sensor is kept in a working state, and the image output by the at least one first sensor is displayed in full screen as the environment image.

[0051] In one possible implementation manner, the driving circuit is further configured to:

[0052] If the total number of image sensors in the image sensor array that are in a working state is greater than 1, stitching a plurality of images output by the image sensor array into a stitched image;

[0053] The brightness and color of the stitched image are adjusted to generate the environment image.

[0054] In one possible implementation manner, the driving circuit is further configured to:

[0055] Selecting an image from the multiple images as a reference image, and stitching two adjacent images starting from the reference image until the stitching of the multiple images output by the image sensor array is completed;

[0056] Among them, the following method is used to stitch two adjacent images:

[0057] Acquire multiple matching points having the same image features from the two adjacent images;

[0058] Calculating the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of the plurality of matching points;

[0059] Calculating the homography matrix of each matching point according to the translation matrix, the rotation matrix and the intrinsic parameter matrix of each matching point;

[0060] Calculating each homography matrix using a minimum median robustness method, selecting matching points corresponding to the homography matrix that meets preset quality requirements to form a best subset, and calculating a final homography matrix based on the best subset so that the two adjacent images become images with the same perspective;

[0061] Aligning the coordinates of the pixels in the images with the same viewing angle, and averaging the pixels in the overlapping portion to obtain a sub-mosaic image of the two adjacent images;

[0062] The stitching process of the two adjacent images is repeated for all the sub-stitched images to obtain the stitched image.

[0063] In one possible implementation manner, the driving circuit is further configured to:

[0064] determining a blurred area in the stitched image according to a change in the optical flow field in the stitched image;

[0065] performing bilateral filtering on the blurred area to obtain a filtered image;

[0066] Performing illumination correction on the filtered image using a preset illumination model;

[0067] The overlapping areas of the two adjacent images corresponding to the blurred area are fused according to the weights to obtain the environment image.

[0068] In one possible implementation manner, the driving circuit is further configured to:

[0069] When the mixed display signal is a dynamic mixed display signal, determining a position of a preset object in the environment image;

[0070] Moving the virtual image of the virtual object in the current frame to the position, and adjusting the angle and size of the virtual image;

[0071] According to the detected lighting of the real environment, the lighting and shadow effects of the virtual image are adjusted to obtain the MR image, and the MR image is transmitted to the display screen for display.

[0072] In one possible implementation manner, the driving circuit is further configured to:

[0073] extracting a plurality of feature points from the environment image;

[0074] Matching the plurality of feature points with feature points of the preset object;

[0075] The position of the successfully matched feature point in the environmental image is determined as the position of the preset object in the environmental image.

[0076] In one possible implementation manner, the driving circuit is further configured to:

[0077] When the mixed display signal is a static mixed display signal, a static virtual image is superimposed and rendered into the environmental image to obtain and display the MR image.

[0078] In a possible implementation manner, the driving circuit is further configured to: when the MR device receives a virtual display signal, turn off the image sensor array and display an image of the virtual world.

[0079] In a possible implementation manner, the driving circuit is further configured to display the environment image upon receiving a real scene display signal.

[0080] In a third aspect, an embodiment of the present disclosure further provides a mixed reality device, including:

[0081] at least one processor, and

[0082] a memory coupled to the at least one processor;

[0083] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method as described in the first aspect above by executing the instructions stored in the memory.

[0084] In a fourth aspect, an embodiment of the present disclosure further provides a readable storage medium, including:

[0085] Memory,

[0086] The memory is used to store instructions. When the instructions are executed by the processor, the device including the readable storage medium performs the method as described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A flowchart of a mixed reality display method provided in an embodiment of the present disclosure;

[0088] Figure 2 A structural diagram of a mixed reality device provided in an embodiment of the present disclosure Figure 1 ;

[0089] Figure 3 A schematic diagram of determining a gaze point according to an embodiment of the present disclosure;

[0090] Figure 4 A schematic diagram of determining a sub-display area that a user is gazing at provided in an embodiment of the present disclosure;

[0091] Figure 5 A schematic diagram of the display area division of a display screen provided in an embodiment of the present disclosure;

[0092] Figure 6 A schematic diagram of an environment image at an initial moment provided by an embodiment of the present disclosure;

[0093] Figure 7 A schematic diagram of an environment image when the gaze duration is less than a preset threshold provided by an embodiment of the present disclosure;

[0094] Figure 8 A schematic diagram of an environment image after the gaze duration exceeds a preset threshold provided by an embodiment of the present disclosure;

[0095] Figure 9 A schematic diagram of obtaining multiple matching points with the same image features in two adjacent images provided by an embodiment of the present disclosure;

[0096] Figure 10 A schematic diagram of image stitching provided in an embodiment of the present disclosure;

[0097] Figure 11 A schematic diagram of a virtual object provided by an embodiment of the present disclosure moving to a position where a preset object is located in an environment image;

[0098] Figure 12 A structural diagram of a mixed reality device provided in an embodiment of the present disclosure Figure 2 ;

[0099] Figure 13 A structural diagram of a mixed reality device provided in an embodiment of the present disclosure Figure 3 . DETAILED DESCRIPTION

[0100] The present disclosure provides a method, a device, and a storage medium for mixed reality display, which are used to solve the technical problems in the prior art of mixed reality display, such as a small field of view and insufficient depth of field.

[0101] In order to better understand the above technical solution, the technical solution of the present disclosure is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical solution of the present disclosure, rather than limitations on the technical solution of the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the technical features in the embodiments can be combined with each other.

[0102] Please refer to Figure 1 and Figure 2 , Figure 1 A flowchart of a mixed reality display method provided in an embodiment of the present disclosure is provided. Figure 2 A structural diagram of a mixed reality device provided in an embodiment of the present disclosure Figure 1 The mixed reality device (MR device) includes: a display screen 1, an eye tracker 2 installed on the display surface 1a of the display screen, and an image sensor array 3 installed on the non-display surface 1b of the display screen, wherein the image sensors in the image sensor array 3 have a corresponding relationship with the multiple sub-display areas included in the display screen 1.

[0103] The image sensors included in the image sensor array 3 can be wide-angle cameras, telephoto cameras, color cameras, black-and-white cameras, depth cameras, and the like. They can be arranged in various ways, such as in a row, in an array of multiple rows and columns, or in two small arrays located at either end of the non-display surface 1b of the display screen. The arrays can be square, matrix, circular, or in other shapes, without limitation. The fields of view of two adjacent image sensors in the image sensor array 3 partially overlap.

[0104] The eye tracker 2 may be an image sensor or an infrared sensor, without limitation.

[0105] It should be noted that Figure 2 It should be understood that it is an MR device for a single eye. If it is an MR device for both eyes, the image sensor array 3 and the eye tracker 2 should be symmetrically distributed on the display screen 1. If each eye corresponds to one eye tracker 2, the two eye trackers are symmetrically distributed on the display surface 1a of the display screen. If there is only one eye tracker 2, the eye tracker should be located in the middle of the display surface 1a of the display screen, and the image sensors in the image sensor array 3 should be symmetrically distributed on the non-display surface 1b of the display screen. Figure 2 The number of image sensors included in the image sensor array 3 does not represent the actual number used, and the installation position of the image sensor array 3 on the non-display surface 1b of the display screen and the installation position of the eye tracker 2 on the front surface 1a of the display screen are not limited to Figure 2 The position shown in the figure can be adjusted as needed in actual use.

[0106] The processing process of the mixed reality display method used in the MR device is as follows:

[0107] Step 101: After receiving the mixed display signal, the user's eyes are tracked by an eye tracker to determine the user's gaze point on the display screen.

[0108] MR devices can include multiple display modes, such as virtual display mode (displaying only virtual world images), real-world display mode (displaying only real-world images), and mixed display mode (displaying both real-world and virtual world images). The display mode of the MR device is set based on the received signal. For example, when a mixed display signal is received, the MR device is set to mixed display mode.

[0109] After receiving the mixed display signal, the user's eyes can be tracked by an eye tracker to determine the user's gaze point on the display screen.

[0110] For example, see Figure 3 This is a schematic diagram of determining the gaze point provided in an embodiment of the present disclosure. The eye tracker can use OpenCV (a cross-platform computer vision and machine learning software library) to track the user's eyes. Figure 3 The eye tracker is not shown in the figure), that is, by capturing the infrared reflection of the user's eyes, the direction of the user's gaze is calculated and defined as the observation vector (such as Figure 3The intersection of the observation vector and the display screen 1 is the user's gaze point on the display screen 1. At the same time, the duration of the eyeball's stay at the same gaze point can also be recorded (denoted as t). When t is greater than 5s, the time parameter T is recorded as 1, otherwise T is 0.

[0111] After the user's gaze point on the display screen is determined, step 102 may be executed.

[0112] Step 102: Determine the sub-display area that the user is gazing at based on the correspondence between the gaze point and the sub-display area in the display screen.

[0113] In the present disclosure, the sub-display area where the gaze point is located can be determined as the sub-display area that the user is gazing at, or the distance between the center point of the sub-display area and the gaze point can be calculated, and the sub-display area with a distance less than the set distance can be determined as the sub-display area that the user is gazing at.

[0114] See Figure 4 A schematic diagram of determining the sub-display area that a user is gazing at provided in an embodiment of the present disclosure.

[0115] exist Figure 4 In the figure, a black dot indicates the gaze point, a hollow circle indicates the center point of the sub-display area, and the dot-dashed line is the dividing line of the sub-display area. Assuming that the set distance is d, by comparing d with d1~d7, it is determined that d1, d3, d4, and d5 are smaller than d, then the sub-display area corresponding to d1, d3, d4, and d5 is determined to be the sub-display area gazed by the user.

[0116] After determining the sub-display area that the user is gazing at, step 103 may be executed.

[0117] Step 103: Determine at least one first image sensor based on the correspondence between the sub-display area and the image sensors in the image sensor array, and adjust the at least one first image sensor to increase the resolution of the image generated by the at least one first image sensor; wherein the distance between the center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and the field of view angles of two adjacent image sensors in the image sensor array partially overlap.

[0118] Determining the first image sensor corresponding to the sub-display area gazed at by the user based on the correspondence between the sub-display areas and the image sensors in the image sensor array may be implemented in the following manner:

[0119] If the image sensors in the image sensor array correspond to the multiple sub-display areas included in the display screen in a one-to-one relationship, the corresponding image sensor is determined to be the first image sensor in combination with the sub-display area that the user is looking at determined in step 102. Figure 4For example, if the sub-display area that the user is gazing at is determined to be the sub-display area corresponding to d1, d3, d4, and d5, then the four image sensors corresponding to the sub-display areas corresponding to d1, d3, d4, and d5 are used as the four first image sensors.

[0120] If the image sensors in the image sensor array and the multiple sub-display areas included in the display screen have a one-to-many correspondence relationship, that is, one image sensor corresponds to at least two sub-display areas, combined with the sub-display area that the user is looking at determined in step 102, the corresponding image sensor can also be determined to be the first image sensor.

[0121] While determining the first image sensor corresponding to the sub-display area that the user is gazing at, at least one first image sensor may also be adjusted in the following manner:

[0122] Determine whether the gaze duration of the gaze point in the sub-display area corresponding to the at least one first image sensor is greater than a preset threshold; if the gaze duration is less than or equal to the preset threshold, increase the resolution of the image generated by the at least one first image sensor to improve the resolution of the corresponding image; if the gaze duration is greater than the preset threshold, keep the at least one first image sensor in a working state, and display the image output by the at least one first sensor as the environmental image in full screen.

[0123] For example, see Figure 5 Schematic diagram of the display area division of the display screen provided in the embodiment of the present disclosure. Assuming that the image sensor array 3 is composed of 3×4 image sensors, the display area of the display screen 1 is divided into 3×4 sub-display areas (the dot-dashed lines are the dividing lines of the sub-display areas), and the 3×4 sub-display areas correspond to the 3×4 image sensors respectively. The gaze point determined by the eye tracker is as follows Figure 5 As shown by the black dot in , the fixation point is located in the sub-display area of the 2nd row and 2nd column of the display screen.

[0124] If the sub-display area where the gaze point is located is determined to be the sub-display area that the user is gazing at, then the sub-display area in the 2nd row and 2nd column of the display screen is the sub-display area that the user is gazing at, and the corresponding sub-display area corresponds to the image sensor in the 2nd row and 2nd column of the image sensor array 3. Therefore, the image sensor in the 2nd row and 2nd column of the image sensor array 3 is determined to be the first image sensor.

[0125] If the sub-display area in which the distance between the gaze point and the center point of the sub-display area is less than the set distance is determined as the sub-display area that the user is gazing at, assuming that the sub-display areas in the 1st row and 2nd column and the 2nd row and 2nd column in the display screen are determined as the sub-display areas that the user is gazing at, then the image sensors in the 1st row and 2nd column and the 2nd row and 2nd column in the image sensor array 3 corresponding to them are both determined as first image sensors.

[0126] See Figure 6-Figure 8 , Figure 6 A schematic diagram of an environment image at an initial moment provided by an embodiment of the present disclosure, Figure 7 A schematic diagram of an environment image when the gaze duration is less than a preset threshold provided by an embodiment of the present disclosure, Figure 8 This is a schematic diagram of an environment image provided by an embodiment of the present disclosure after the gaze duration is greater than a preset threshold. Figure 6-Figure 7 The middle fixation points are all located in the sub-display area of the second row and second column of the display screen.

[0127] At the initial moment, the environment image displayed on the display is as follows Figure 6 As shown in FIG, assuming that the gaze point stays in the sub-display area corresponding to the first image sensor (i.e., the sub-display area in the second row and second column of the display screen) for 3 seconds and the preset threshold is 5 seconds, if the gaze time is less than the preset threshold, the resolution of the first image sensor (the image sensor corresponding to the sub-display area in the second row and second column) is increased so that the part of the image corresponding to the first image sensor in the environmental image is displayed in high definition. At this time, the environmental image displayed on the display screen is as follows: Figure 7 shown.

[0128] Assume that the fixation time of the gaze point in the sub-display area corresponding to the first image sensor is 6 seconds. Therefore, the fixation time is longer than the preset threshold. At this time, only the first image sensor is kept in the working state, and the image output by the first image sensor is displayed as the environment image in full screen. At this time, the environment image displayed on the display screen is as follows: Figure 8 shown.

[0129] It should be noted that if the sub-display area that the user is looking at includes multiple sub-display areas, similar operations are performed on the first image sensors corresponding to these sub-display areas. The difference is that the images output by these first image sensors are spliced before being displayed. To save space, we will not go into details one by one. Figure 6-Figure 8 In order to facilitate observation, the dividing lines of each sub-area in the display screen are shown as dotted lines. In actual application, the dividing lines will not be displayed. Figure 6-Figure 8 The high-resolution part of the image is indicated by a solid line, and the low-resolution part is indicated by a dotted line. Figure 6-Figure 8The environment image is an image obtained by splicing and fusing multiple images output by the image sensor array. The specific processing method is introduced in step 103.

[0130] After the first image sensor is adjusted, step 104 may be performed.

[0131] Step 104: Overlay and render the environment image output by the image sensor array and the virtual image to obtain and display an MR image.

[0132] Before superimposing and rendering the environment image output by the image sensor array with the virtual image, it also includes:

[0133] If the total number of image sensors in the image sensor array that are in a working state is greater than 1, multiple images output by the image sensor array are stitched into a stitched image; and the brightness and color of the stitched image are adjusted to generate an environment image.

[0134] To stitch multiple images output by the image sensor array into a single stitched image, an image stitching algorithm can be used in real time to stitch together multiple images output by the image sensor array at the same time. Because the fields of view of adjacent image sensors in the image sensor array overlap, stitching multiple images in real time using an image stitching algorithm can prevent a fragmented image.

[0135] The following methods can be used to stitch multiple images output by the image sensor array into a stitched image:

[0136] An image is selected from the multiple images as a reference image, and two adjacent images are spliced starting from the reference image until the splicing of the multiple images output by the image sensor array is completed.

[0137] To implement the real-time image stitching algorithm, the image sensors in the image sensor array need to be calibrated before shooting, such as calibrating the installation position and setting shooting parameters. This ensures greater consistency in the images captured by each image sensor, facilitating subsequent image stitching. During stitching, if the image sensor array simultaneously outputs images captured by each image sensor, the image captured by the image sensor located in the image sensor array corresponding to the center of the display screen can be used as the reference image. If the image sensor array outputs images captured by some image sensors, the image captured by the first image sensor corresponding to the sub-pixel area where the gaze point is located can be used as the reference image. During the stitching process, two adjacent images are stitched together starting from the reference image until the stitching of multiple images output by the image sensor array is completed.

[0138] To stitch two adjacent images together, you can use the following methods:

[0139] Multiple matching points with the same image features are obtained from two adjacent images; the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of the multiple matching points are calculated; the homography matrix of each matching point is calculated based on the translation matrix, rotation matrix, and intrinsic parameter matrix of each matching point; each homography matrix is calculated using the minimum median robustness method, and the matching points corresponding to the homography matrix that meets the preset quality requirements are selected to form the best subset. The final homography matrix is calculated based on the best subset to transform the two adjacent images into images with the same perspective; the pixels in the images with the same perspective are aligned, and the pixels of the overlapping parts are averaged to obtain a sub-mosaic image of the two adjacent images.

[0140] See Figure 9 A schematic diagram of obtaining multiple matching points with the same image features in two adjacent images provided by the embodiment of the present disclosure. Figure 9 There is a small part of the road that the vehicle is traveling on that is common to both images. By detecting the feature points in the two images and matching them, we can obtain multiple matching points ( Figure 9 The method takes the feature points on the boundary line of the road in the middle as an example, represented by a circle with a slash, and calculates the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of these matching points; calculates the homography matrix of each matching point based on the translation matrix, rotation matrix, and intrinsic parameter matrix of each matching point; then calculates each homography matrix using the minimum median robustness method, selects the matching points corresponding to the homography matrix that meets the preset quality requirements to form the best subset, calculates the final homography matrix based on the best subset, and transforms the two adjacent images into images with the same perspective; aligns the pixels in the images with the same perspective, and averages the pixels in the overlapping parts to obtain a sub-mosaic of the two adjacent images.

[0141] The above processing involves converting image coordinates into world coordinates and image fusion. Image coordinate transformation can match and filter image feature points with fixed characteristics such as illumination and orientation in multiple images, and calculate the image captured by each image sensor (the image captured by the i-th image sensor in the image sensor array is denoted as P i ) and record the focal length from the optical center of the i-th image sensor to the imaging plane (denoted as d i ), and then transform each image from the camera coordinate system to the world coordinate system, and then calculate the homography matrix of each image so that all images have the same perspective in the same coordinate system to complete image stitching.

[0142] The image is transformed from the camera coordinate system to the world coordinate system mainly through the translation matrix T i , rotation matrix Ri Translate and rotate the image. Then use T i 、R i d i Calculate image P i The projection transformation matrix Q i =T i ·F i ·R i , through Q i Description i The translation and rotation transformation from the camera coordinate system to the world coordinate system.

[0143] in,

[0144] Then, use Q i Calculate P i The homography matrix H i =inv(Q i )·Q r , used to represent P i With P r The coordinate correspondence between each pixel point during image matching, where inv(Q i ) is the matrix Q i The inverse matrix of . After homography transformation, all images become unified perspective images U i , and then U i The coordinate values of each pixel in the image are converted into homogeneous coordinates, so that the pixels of all images are moved to the same coordinate system to form a complete image. Then the values of the overlapping pixels are averaged to obtain the sub-mosaic image.

[0145] For example, Figure 9 The two images in the image are transferred to the same coordinate system and the same viewing angle for splicing, and the average value of the overlapping area is calculated to obtain the following Figure 10 The last part of the stitched image is shown in Figure 10 A schematic diagram of image stitching provided in an embodiment of the present disclosure.

[0146] The stitching process of two adjacent images is repeated for all sub-stitching images to obtain a stitched image.

[0147] When performing image stitching, since two adjacent images are used, and these two adjacent images have different fields of view, and some of the fields of view are overlapping, the two fields of view are used to calculate the change matrix (i.e., translation matrix, rotation matrix, etc.) and camera intrinsic parameters (i.e., intrinsic parameter matrix) during the stitching process, which can achieve faster processing speed. In the processing process, the minimum median robustness method is used to select the best subset, and then the best subset is used to calculate the initialization estimate and mask of the homography matrix, further reducing the reprojection error and reducing the noise.

[0148] Since there are some blurred areas in the stitched image (such as unclear object outlines, uneven exposure, etc.), these blurred areas need to be processed to make the entire stitched image consistent in overall chromaticity and brightness, and thus obtain the environmental image.

[0149] In one possible implementation, adjusting the brightness and color of the stitched image to generate the environment image can be achieved by:

[0150] According to the changes in the optical flow field in the stitched image, the blurred area contained in the stitched image is determined; the blurred area is bilaterally filtered to obtain a filtered image; the filtered image is subjected to illumination correction using a preset illumination model; the overlapping areas of the two adjacent images corresponding to the blurred area are fused according to the weights to obtain an environmental image.

[0151] For example, the horizontal optical flow in the stitched image can be approximated as the inverse of the depth (parallax). A normalized sum of squared errors is retrieved and calculated for the area near the blurred area in the stitched image. The initial optical flow is calculated based on the block optical flow and its confidence map. This is then processed through bilateral filtering to make it spatially continuous with the surrounding optical flow, ultimately resulting in an optical flow field with clear boundaries and uniform and smooth exposure. The camera's illumination model is then used to correct illumination non-uniformities within the filtered image. A histogram mapping table is established between the two adjacent images based on the relationship between the overlapping areas of the two adjacent images in the stitched image. The overlapping areas of the two adjacent images corresponding to the blurred area are then fused according to the weights of the mapping table (i.e., an overall mapping transformation is performed), ultimately resulting in an environmental image with consistent overall brightness and color.

[0152] Assume that the two images corresponding to a certain blurred area in the filtered image are Image 1 and Image 2. Let p and t be the starting addresses of the i-th row pixels of Image 1 and Image 2 after stitching respectively, d be the address of the first pixel in the i-th row of the filtered image, row be the row of d, where 0 ≤ i < row, col be the column of p, 0 ≤ j < col, alpha be the pixel weight coefficient of Image 1, start be the starting position of the overlapping area, and width be the width of the overlapping area. When there are black dots representing no pixels in t, alpha = 1 and d completely copies the data in p; otherwise, alpha = (width - (j - start)) / width. The fusion formula is as follows:

[0153] d[j×3] = p[j*3]×alpha + t[j×3]*(1 - alpha);

[0154] d[j×3 + 1] = p[j×3 + 1]×alpha + t[j×3 + 1]×(1 - alpha);

[0155] d[j×3 + 2] = p[j×3 + 2]×alpha + t[j×3 + 2]×(1 - alpha).

[0156] Through the above method, the pixels in the overlapping area can be fused to obtain the environmental image.

[0157] By using the image sensor array in the embodiments of the present disclosure to capture and stitch and fuse images in the real environment, the obtained image has a larger viewing angle, which can exceed the human eye limit, enabling users to see farther and more delicate scenes, thus completely exceeding the viewing angle of existing MR devices and greatly enhancing the user immersion experience.

[0158] After obtaining the environmental image, it is also necessary to superimpose and render the environmental image output by the image sensor array with the virtual image to obtain and display the MR image, which can be implemented in the following manner:

[0159] When the mixed display signal is a dynamic mixed display signal, determine the position of the preset object in the environmental image; move the virtual image of the virtual object in the current frame to the position of the preset object in the environmental image, and adjust the angle and size of the virtual image; according to the detected illumination of the real environment, adjust the illumination and shadow effects of the virtual image to obtain the MR image, and transmit the MR image to the display screen for display.

[0160] A possible implementation manner to determine the position of the preset object in the environmental image can be achieved by the following method:

[0161] Extract multiple feature points from the environment image; match the multiple feature points with feature points of a preset object; and determine the position of the successfully matched feature points in the environment image as the position of the preset object in the environment image.

[0162] For example, Figure 7 For example, the preset object is the head of the vehicle, and the virtual object is a compass. Multiple feature points are extracted from the environmental image. These feature points are usually fixed, such as the corners of the vehicle body, license plate, and headlights. Multiple feature points representing the head of the vehicle can be pre-stored and matched with multiple feature points obtained from the environmental image. After the match is successful, the position of the head of the vehicle in the environmental image can be determined, and then the virtual image of the virtual object in the current frame can be moved to the position of the preset object in the environmental image, and the angle and size of the virtual image can be adjusted, such as Figure 11 , which is a schematic diagram of a virtual object provided by an embodiment of the present disclosure moving to the position of a preset object in an environment image.

[0163] After receiving the dynamic mixed display signal, the foreground information (environmental image) transmitted by the image sensor is used to identify the scene where the current user is located using an algorithm, and then different dynamic virtual images are intelligently rendered according to the scene. If it is recognized that the user is in a city street, a dynamic scene of a virtual dinosaur flying by can be rendered. This can present an overall picture with a stronger sense of space, a wide field of view, and delicate picture quality.

[0164] In addition, the above-mentioned image processing related algorithms (such as image stitching algorithms, rendering algorithms, etc.) can be integrated into the driving circuit of the display screen, which can speed up the image processing speed, enhance the rendering effect, and bring a more realistic visual experience to users.

[0165] In one possible implementation, the environmental image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image, including:

[0166] When the mixed display signal is a static mixed display signal, the static virtual image is superimposed and rendered into the environment image to obtain and display the MR image.

[0167] For example, when a circuit engineer is soldering a circuit, the image sensor array can be used to magnify the soldering point (the soldering point where the user is looking), and at the same time, a virtual circuit schematic diagram can be superimposed on the environmental image composed of the soldering points to form a static MR image displayed on the display screen. In this way, the user can conveniently modify the circuit by directly viewing the display screen.

[0168] In one possible implementation, when a virtual display signal is received, the image sensor array is turned off and the image of the virtual world is displayed.

[0169] In a possible implementation, when a real scene display signal is received, the environment image is displayed.

[0170] To facilitate rapid switching of MR devices between the aforementioned display modes while maintaining high-speed processing, the images of the rendered virtual world can be processed in the application processor (AP), and the image signal processing (ISP) that processes the images output by the image sensor array can be integrated into the display driver circuit. Switching between different display modes can be achieved with a single-touch switch using physical buttons set in the AP. Figure 12 A structural diagram of a mixed reality device provided in an embodiment of the present disclosure Figure 2 .

[0171] Figure 12 The AP includes physical buttons, a general-purpose input / output (GPIO) interface, a refresh register, a Mobile Industry Processor Interface (MIPI) DSI (a serial interface used in display technology), and a graphics processing unit (GPU). The GPU is used to render and generate images of the virtual world and transmit them to the random access memory (RAM) in the driver via the MIPI DSI. The physical buttons transmit the control information input by the user to the refresh register and the driver circuit via the GPIO interface. If the control information indicates the use of a virtual display mode, after the control information is transmitted to the driver circuit, the driver circuit generates a corresponding control signal to control the image sensor array to turn off, otherwise it turns on. If the control information indicates the use of a real-scene display mode, after the control information is transmitted to the refresh register, the MIPI DSI stops outputting the virtual world image rendered by the GPU to the RAM in the driver circuit. In this way, users can independently choose to experience a mixed display world or a purely virtual world through the physical buttons.

[0172] Figure 12 The driving circuit in the image sensor array can generate a control signal to control the image sensor in the image sensor array to be turned off or on, and includes a MIPI camera serial interface (CSI) interface for receiving the image signal output by the image sensor array, and an ISP for processing the image signal received from the MIPI CSI interface. The ISP sends the processed signal to the RAM, and the algorithm IP processes the data in the RAM.

[0173] It should be noted that those skilled in the art can also Figure 12 The components of the content shown may be modified, such as integrating the AP part into the driving circuit, or integrating the physical buttons and GPIO into the driving circuit, or other similar modifications. These changes and modifications do not depart from the spirit and scope of the present disclosure.

[0174] Based on the same inventive concept, an embodiment of the present disclosure provides a mixed reality device. The specific implementation of the mixed reality display method of the mixed reality device can be found in the description of the method embodiment part. The repeated parts will not be repeated. Figure 13 , the mixed reality device includes:

[0175] A display screen 1301, an eye tracker 1302 mounted on a display surface of the display screen 1301, and an image sensor array 1303 mounted on a non-display surface of the display screen 1301; wherein the image sensors in the image sensor array correspond to the plurality of sub-display areas included in the display screen;

[0176] The driving circuit 1304 is configured to, after receiving the mixed display signal, control the eye tracker 1302 to track the user's eyes to determine the user's gaze point on the display screen 1301; determine at least one first image sensor corresponding to the gaze point based on the correspondence between the gaze point and the sub-display area in the display screen, and the correspondence between the sub-display area and the image sensors in the image sensor array, and adjust the at least one first image sensor to increase the resolution of the at least one first image sensor; wherein the distance between the center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and the field of view angles of two adjacent image sensors in the image sensor array partially overlap; and superimpose and render the environmental image output by the image sensor array 1303 and the virtual image to obtain and display the MR image.

[0177] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0178] Determining whether a gaze time duration during which the gaze point stays in the sub-display area corresponding to the at least one first image sensor is greater than a preset threshold;

[0179] If the gaze duration is less than or equal to the preset threshold, increasing the resolution of the image generated by the at least one first image sensor to improve the resolution of the corresponding image;

[0180] If the gaze duration is greater than the preset threshold, the at least one first image sensor is kept in a working state, and the image output by the at least one first sensor is displayed in full screen as the environment image.

[0181] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0182] If the total number of image sensors in the image sensor array that are in working state is greater than 1, stitching the multiple images output by the image sensor array 1303 into a stitched image;

[0183] The brightness and color of the stitched image are adjusted to generate the environment image.

[0184] In one possible implementation manner, the driving circuit is further configured to:

[0185] Selecting an image from the multiple images as a reference image, and stitching two adjacent images starting from the reference image until the stitching of the multiple images output by the image sensor array is completed;

[0186] Among them, the following method is used to stitch two adjacent images:

[0187] Acquire multiple matching points having the same image features from the two adjacent images;

[0188] Calculating the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of the plurality of matching points;

[0189] Calculating the homography matrix of each matching point according to the translation matrix, the rotation matrix and the intrinsic parameter matrix of each matching point;

[0190] Calculating each homography matrix using a minimum median robustness method, selecting matching points corresponding to the homography matrix that meets preset quality requirements to form a best subset, and calculating a final homography matrix based on the best subset so that the two adjacent images become images with the same perspective;

[0191] Aligning the coordinates of the pixels in the images with the same viewing angle, and averaging the pixels in the overlapping portion to obtain a sub-mosaic image of the two adjacent images;

[0192] The stitching process of the two adjacent images is repeated for all the sub-stitched images to obtain the stitched image.

[0193] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0194] determining a blurred area in the stitched image according to a change in the optical flow field in the stitched image;

[0195] performing bilateral filtering on the blurred area to obtain a filtered image;

[0196] Performing illumination correction on the filtered image using a preset illumination model;

[0197] The overlapping areas of the two adjacent images corresponding to the blurred area are fused according to the weights to obtain the environment image.

[0198] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0199] When the mixed display signal is a dynamic mixed display signal, determining a position of a preset object in the environment image;

[0200] Moving the virtual image of the virtual object in the current frame to the position, and adjusting the angle and size of the virtual image;

[0201] According to the detected lighting of the real environment, the lighting and shadow effects of the virtual image are adjusted to obtain the MR image, and the MR image is transmitted to the display screen 1301 for display.

[0202] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0203] extracting a plurality of feature points from the environment image;

[0204] Matching the plurality of feature points with feature points of the preset object;

[0205] The position of the successfully matched feature point in the environmental image is determined as the position of the preset object in the environmental image.

[0206] In one possible implementation manner, the driving circuit 1304 is further configured to:

[0207] When the mixed display signal is a static mixed display signal, a static virtual image is superimposed on the environmental image to obtain and display the MR image.

[0208] In a possible implementation manner, the driving circuit 1304 is further configured to: when the MR device receives a virtual display signal, turn off the image sensor array and display the image of the virtual world.

[0209] In a possible implementation manner, the driving circuit 1304 is further configured to display the environment image upon receiving the real scene reality.

[0210] Based on the same inventive concept, an embodiment of the present disclosure provides a mixed reality device, comprising: at least one processor, and

[0211] a memory coupled to the at least one processor;

[0212] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the mixed reality display method as described above by executing the instructions stored in the memory.

[0213] Based on the same inventive concept, the present disclosure further provides a readable storage medium, including:

[0214] Memory,

[0215] The memory is used to store instructions, and when the instructions are executed by the processor, the device including the readable storage medium completes the mixed reality display method as described above.

[0216] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0217] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0220] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A mixed reality display method, applied to a driving circuit in an MR device, wherein: The MR device also includes a display screen, an eye tracker installed on the display surface of the display screen, an image sensor array installed on the non-display surface of the display screen, and an application chip. The application chip includes a physical button, a universal input and output interface, a refresh memory, a mobile industry processor interface, and a graphics processor. The output end of the physical button is connected to the input end of the universal input and output interface, one output end of the universal input and output interface is connected to an input end of the drive circuit, the other output end of the universal input and output interface is connected to the input end of the refresh memory, the output end of the refresh memory is connected to an input end of the mobile industry processor interface, the output end of the graphics processor is connected to another input end of the mobile industry processor, and the output end of the mobile industry processor is connected to the drive circuit; the physical button is connected to the input end of the universal input and output interface. The method comprises generating control information based on a user's operation, wherein the control information is used to indicate a display mode of the MR device; the general input / output interface is used to transmit the control information to the refresh memory, and the refresh memory is used to transmit the control information to the mobile industry processor interface; the graphics processor is used to generate a virtual image and transmit the virtual image to the mobile industry processor interface; when the control information indicates that the display mode is a real scene display mode, the mobile processor interface stops outputting the virtual image to the drive circuit; if the control information indicates that the display mode is a virtual display mode, the drive circuit controls the image sensor array to be turned off; wherein the image sensors in the image sensor array have a corresponding relationship with the multiple sub-display areas included in the display screen, and two adjacent sub-display areas are connected and do not overlap. The method comprises: If the control information indicates that the display mode is a mixed display mode, after receiving a mixed display signal corresponding to the mixed display mode from the universal input / output interface, tracking the user's eyes by the eye tracker to determine the user's gaze point on the display screen; determining the sub-display area gazed by the user according to a correspondence between the gaze point and the sub-display area in the display screen; determining at least one first image sensor based on a correspondence between the sub-display areas and image sensors in the image sensor array, and adjusting the at least one first image sensor to increase a resolution of an image generated by the at least one first image sensor; wherein a distance between a center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and two adjacent image sensors in the image sensor array have different field of view angles and partially overlap; The environment image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image; wherein the environment image has no sense of fragmentation.

2. The method according to claim 1, wherein Adjusting the at least one first image sensor includes: Determining whether a gaze time duration during which the gaze point stays in the sub-display area corresponding to the at least one first image sensor is greater than a preset threshold; If the gaze duration is less than or equal to the preset threshold, increasing the resolution of the image generated by the at least one first image sensor to improve the resolution of the first image; If the gaze duration is greater than the preset threshold, the at least one first image sensor is kept in a working state, and the image output by the at least one first sensor is displayed in full screen as the environment image.

3. The method according to claim 2, wherein: Before superimposing and rendering the environment image output by the image sensor array with the virtual image, the method further includes: If the total number of image sensors in the image sensor array that are in a working state is greater than 1, stitching a plurality of images output by the image sensor array into a stitched image; The brightness and color of the stitched image are adjusted to generate the environment image.

4. The method according to claim 3, wherein: Stitching a plurality of images output by the image sensor array into a stitched image, comprising: Selecting an image from the multiple images as a reference image, and stitching two adjacent images starting from the reference image until the stitching of the multiple images output by the image sensor array is completed; Among them, the following method is used to stitch two adjacent images: Acquire multiple matching points having the same image features from the two adjacent images; Calculating the translation matrix, rotation matrix, and intrinsic parameter matrix corresponding to each of the plurality of matching points; Calculating the homography matrix of each matching point according to the translation matrix, the rotation matrix and the intrinsic parameter matrix of each matching point; Calculating each homography matrix using a minimum median robustness method, selecting matching points corresponding to the homography matrix that meets preset quality requirements to form a best subset, calculating a final homography matrix based on the best subset, and transforming the two adjacent images into images with the same perspective; Aligning the coordinates of the pixels in the images with the same viewing angle, and averaging the pixels in the overlapping portion to obtain a sub-mosaic image of the two adjacent images; The stitching process of the two adjacent images is repeated for all the sub-stitched images to obtain the stitched image.

5. The method according to claim 4, wherein: Adjusting the brightness and color of the stitched image to generate the environment image includes: determining a blurred area in the stitched image according to a change in the optical flow field in the stitched image; performing bilateral filtering on the blurred area to obtain a filtered image; Performing illumination correction on the filtered image using a preset illumination model; The overlapping areas of the two adjacent images corresponding to the blurred area are fused according to the weights to obtain the environment image.

6. The method according to any one of claims 2 to 5, wherein: The method comprises superimposing and rendering the environment image output by the image sensor array and the virtual image to obtain and display an MR image, including: When the mixed display signal is a dynamic mixed display signal, determining a position of a preset object in the environment image; Moving the virtual image of the virtual object in the current frame to the position, and adjusting the angle and size of the virtual image; According to the detected lighting of the real environment, the lighting and shadow effects of the virtual image are adjusted to obtain the MR image, and the MR image is transmitted to the display screen for display.

7. The method according to claim 6, wherein: Determining a position of a preset object in the environment image includes: extracting a plurality of feature points from the environment image; Matching the plurality of feature points with feature points of the preset object; The position of the successfully matched feature point in the environmental image is determined as the position of the preset object in the environmental image.

8. The method according to any one of claims 2 to 5, wherein: The method comprises superimposing and rendering the environment image output by the image sensor array and the virtual image to obtain and display an MR image, including: When the mixed display signal is a static mixed display signal, a static virtual image is superimposed and rendered into the environmental image to obtain and display the MR image.

9. The method of claim 1, wherein: When a virtual display signal is received, the image sensor array is turned off and the image of the virtual world is displayed.

10. The method according to any one of claims 2 to 5, wherein: When a real scene display signal is received, the environment image is displayed.

11. A mixed reality device, wherein: include: A display screen, an eye tracker mounted on a display surface of the display screen, and an image sensor array mounted on a non-display surface of the display screen; wherein the image sensors in the image sensor array correspond to a plurality of sub-display areas included in the display screen, and two adjacent sub-display areas are connected and do not overlap; An application chip, comprising a physical button, a universal input / output interface, a refresh memory, a mobile industry processor interface, and a graphics processor, wherein the output end of the physical button is connected to the input end of the universal input / output interface, one output end of the universal input / output interface is connected to an input end of a drive circuit, another output end of the universal input / output interface is connected to the input end of the refresh memory, the output end of the refresh memory is connected to an input end of the mobile industry processor interface, the output end of the graphics processor is connected to another input end of the mobile industry processor, and the output end of the mobile industry processor is connected to the drive circuit; the physical button is used to generate control information based on user operations, and the control information is used to indicate the display mode of the mixed reality device; the universal input / output interface is used to transmit the control information to the refresh memory, and the refresh memory is used to transmit the control information to the mobile industry processor interface; the graphics processor is used to generate a virtual image and transmit it to the mobile industry processor interface; and when the control information indicates that the display mode is a real-scene display mode, the mobile processor interface stops outputting the virtual image to the drive circuit; The driving circuit is configured to, when the control information indicates that the display mode is a virtual display mode, turn off the image sensor array; if the control information indicates that the display mode is a hybrid display mode, then, after receiving a hybrid display signal corresponding to the hybrid display mode from the universal input / output interface, control the eye tracker to track the user's eyes to determine the user's gaze point on the display screen; and, based on a correspondence between the gaze point and a sub-display area in the display screen, and a correspondence between the sub-display area and image sensors in the image sensor array, determine at least one first image sensor corresponding to the gaze point, and adjust the at least one first image sensor to increase the resolution of an image generated by the at least one first image sensor; wherein the distance between a center point of the sub-display area corresponding to the first image sensor and the gaze point is less than a set distance, and the field of view angles of two adjacent image sensors in the image sensor array are different and partially overlap; and the environmental image output by the image sensor array is superimposed and rendered with the virtual image to obtain and display an MR image; wherein the environmental image has no sense of fragmentation.

12. A mixed reality device, wherein: include: at least one processor, and a memory coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method according to any one of claims 1 to 10 by executing the instructions stored in the memory.

13. A readable storage medium, wherein: Including memory, The memory is used to store instructions. When the instructions are executed by the processor, the device including the readable storage medium performs the method according to any one of claims 1 to 10.

Images