Method for processing environmental image data in video see-through (VST), head-mounted display device, and storage medium

Abstract

The method of processing environment image data, and products related thereto, are applicable to extended reality head-mounted display having at least two cameras to capture real world images which are displayed to user through display screens. In response to user's zoom-in command, a target pixel T is determined based on a vertical perpendicular distance Z between the target pixel T and the mixed reality head-mounted display, and then calculating Xold; a new vertical perpendicular distance Znew is obtained after target pixel T is zoomed in, and then calculating Xnew; magnifying the images on the display screens by (Xnew / Xold) times, and then translating each pixel of the magnified images on the display screens simultaneously by a same distance as how parallax viewing positions on the display screens translate when Xold is changed to Xnew when the target pixel T is zoomed in.

Description

(19) *EP004730765A1* (11) EP 4 730 765 A1 (12) EUROPEAN PATENT APPLICATION (43) Date of publication: 22.04.2026 Bulletin 2026 / 17 (21) Application number: 25191202.8 (22) Date of filing: 23.07.2025 (51) International Patent Classification (IPC): H04N 13 / 128 (2018.01) H04N 13 / 139 (2018.01) H04N 13 / 239 (2018.01) H04N 13 / 344 (2018.01) H04N 23 / 69 (2023.01) (52) Cooperative Patent Classification (CPC): H04N 13 / 239; G02B 27 / 017; H04N 13 / 128; H04N 13 / 139; H04N 13 / 344; H04N 23 / 69; G02B 2027 / 0138; G02B 2027 / 014 (84) Designated Contracting States: AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR Designated Extension States: BA Designated Validation States: GE KH LA MA MD TN (30) Priority: 21.10.2024 CN 202411464337 (71) Applicant: Dalian Situne Technology Co., Ltd Dalian City, Liaoning Province 116023 (CN) (72) Inventor: PAN, JONG-GUANG Beijing, 100102 (CN) (74) Representative: Cabinet Chaillot 16 / 20, avenue de l’Agent Sarre B.P. 74 92703 Colombes Cedex (FR) (54) METHOD FOR PROCESSING ENVIRONMENTAL IMAGE DATA IN VIDEO SEE‑THROUGH (VST), HEAD‑MOUNTED DISPLAY DEVICE, AND STORAGE MEDIUM (57) The method of processing environment image data, and products related thereto, are applicable to extended reality head-mounted display having at least two cameras to capture real world images which are displayed to user through display screens. In response to user’s zoom-in command, a target pixel T is deter- mined based on a vertical perpendicular distance Z be- tween the target pixel T and the mixed reality head- mounted display, and then calculatingXold; a newvertical perpendicular distance Znew is obtained after target pixel T is zoomed in, and then calculatingXnew;magnifying the images on the display screens by (Xnew / Xold) times, and then translatingeachpixel of themagnified imageson the display screens simultaneously by a same distance as how parallax viewing positions on the display screens translate when Xold is changed to Xnew when the target pixel T is zoomed in. EP 4 73 0 76 5 A 1 Processed by Luminess, 75001 PARIS (FR) Description TECHNICAL FIELD

[0001] The present invention belongs to the field of video see-through technology, specifically relating to a method for processingenvironmental imagedata invideosee-through (VST),ahead-mounteddisplaydevice,andastoragemedium. BACKGROUND OF THE INVENTION

[0002] Extended reality (XR) glasses are divided into two categories: AR (augmented reality) glasses and VR (virtual reality) glasses. ARglasses are usually implemented by usingOptical See-Through (OST) technology or optical lenses to view the surrounding environment (hereinafter referred to as optical see-through OST). VR glasses are purely virtual devices that cannot view the external environment. In recent years, MR (mixed reality) glasses have emerged, where cameras are used in VR glasses to view the surroundings through Video See-Through (VST) technology (or alternatively referred to as Visual Pass-Through technology). Since OST does not capture environmental images or video streams, it cannot processenvironmental videosor images. In contrast, VSTusescameras to captureenvironmental imagesor video streams, allowing computational processing and post-processing of the surrounding environment in the captured images or video streams. Beyond rendering custom virtual objects and backgrounds as display contents, a metaverse further requires the ability to post-process the surrounding environment in the captured images or video streams.

[0003] On traditional tablets andsmartphones, users can freely zoom inandout of imagesor video streamsbyusing two fingers. Features such as zoom in / out or adjusting the focus distance (f) are already available during video recording.With technological advancements, MR glasses will gradually become extensions of human vision, enabling users to magnify distantor small objects.On2Dscreens, zoom in / out functionsareeasily implementedbecause theydonot involveparallax issues-zooming in andout equate to simply scaling images or video streams.However, in a pair ofMRglasses, zooming in andout isnot simplyscaling imagesorvideostreams.Whilescaling imagesorvideostreams isstraightforward, zooming in and out require adjusting the parallax between the left and right eyes. Currently, traditional MR glasses lack methods for post-processing environmental image data in VST to achieve zoom in / out effects. BRIEF SUMMARY OF THE INVENTION

[0004] It is an object of the present invention is to provide a method for processing environmental image data in video see-through (VST), ahead-mounteddisplaydevice,andastoragemedium.Thepresent invention isapplicable tosystems of MR (Mixed Reality) head-mounted display devices. Based on user’s interactive commands, the method of the present invention performs post-processing of the environmental image data in VST, enabling immersive viewing of the surrounding environment being zoomed in or out through VST. This allows users to magnify and observe distant or very small objects, and achieves the visual perception of objects being zoomed in or out.

[0005] Amethod forprocessingenvironmental imagedata invideosee-through,applicable toasystemofamixed reality head-mounted display; at least two cameras of the system corresponding to two eyes of a user capture real environment images, which are displayed and viewed on a left screen and a right screen of the mixed reality head-mounted display; based on user’s interactive commands, the systempost-processes the real environment images to create visual effects of zooming in / out of objectswithin the real environment images; the interactive commands includepreset zoom-inandzoom- out commands; the system predicts a zoomdistance based on the interactive commands; the system post-processes the real environment images according to the following steps: step 1: in response to a zoom-in command given by the user who views the real environment images through VST, determining a target pixel T based on a distanceZwhich is a vertical perpendicular distance between the target pixel Tand themixed reality head-mounted display, wherein a pixel closest to one endof said distanceZopposite another end thereof at the mixed reality head-mounted display is defined as the target pixel T; step 1 comprises the following steps: assuming the left screen and the right screen of themixed reality head-mounted display align pixels based on field of view (FOV)ofa left cameraanda right cameraof thesystemrespectively, a total numberof pixelsof an imagecaptured bya respectiveoneof the left cameraand right camera is equal to a total number of screenpixels of a respectiveoneof the left screen and the right screen either with or without correction; given that a total number of pixels along anX-axis of a respective one of the left screen and the right screen is Xtotal, the FOVof the left camera and the right camera is known, an interpupillary distance (DIPD) between the left camera and the right camera is known, and let a pixel per degree (PPD) beingPPD=Xtotal / FOV; using point L to represent a center point of the left screen and a center point of the left camerasimultaneously, andusingpointR to representacenterpoint of the right screenandacenterpointof the right camera simultaneously, a line connecting said point L and said pointR or a line parallel to the line connecting said point L and said point R is defined as the X-axis; establishing XY coordinate systems on the left screen and the right 2 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 screen respectively, with said point L and said point R being origins of the XY coordinate systems respectively; wherein on the left screen, a distance on an imaging plane of the left screen between a point corresponding to perpendicular projection of said point L on the imaging plane of the left screen and a parallax viewing position according towhich the target pixel T is viewed is definedasXL, andon the right screen, a distanceonan imaging plane of the right screen between a point corresponding to perpendicular projection of said point R on the imaging plane of the right screen and a parallax viewing position according towhich the target pixel T is viewed is defined asXR; values of said XL and said XR are represented on the left screen and on the right screen respectively as pixel values; as an object asviewed from the left screenand the right screen is zoomed in, theparallax viewingpositionson the left screen and the right screen move along or parallel to the X-axis symmetrically from said point L and said point R towards a center of the mixed reality head-mounted display; a normal line passing perpendicularly through the X-axis at said point L is defined as a left central line, and a normal line passing perpendicularly through the X-axis at said point R is defined as a right central line; an angle between the left central line and a line passing through said point L and the parallax viewing position on the left screen according to which the target pixel T is viewed as defined as θL; an angle between the right central line and a line passing through said point R and the parallax viewing position on the right screen according towhich the target pixel T is viewed as defined as θR; in the real world, D is used to represent values of howmuchaperpendicular projectionpoint of the target pixel Tonto theX-axis is distanced fromsaid point Landsaid pointR,wherein anormal lineextending from the target pixel Tand intersecting perpendicularlywith theX-axis divides D into DL andDR, wherein DL is a distance between said point L and an intersection point of the X-axis and the normal line extending from the target pixel Tand intersecting perpendicularly with the X-axis, and DR is a distance between said point R and the intersection point of the X-axis and the normal line extending from the target pixel T and intersecting perpendicularly with the X-axis; defining DL+DR=DIPD, wherein DIPD is the interpupillary distance between the left camera and the right camera and also a fixed known distance between said point L and said point R; during zoom in / out process of the target pixel T, DL and DR remain unchanged; scanning row by row a left image and a right image captured by the left camera and the right camera respectively, comparing all pixels in each row to find pixels in the left image and the right image with identical depth information or grayscale colors and consistent positional arrangements; obtaining values of XL and XR of each of these pixels, and then calculate for each of these pixels: and as well as the distance Z according to the following formula: defining thepixel closest to saidoneendof saiddistanceZopposite saidanother end thereof at themixed reality head- mounted display as the target pixel T, which is currently at an original position; said target pixel T at said original position is defined as Told; saving original position data of Told, wherein said original position data of Told includes a value of XL corresponding to Told , definedasXold-left , a value of θL corresponding to Told, definedas θold-left , a value of XR corresponding to Told , defined as Xold-right , a value of θR corresponding to Told, defined as θold-right , and a value of Zold , which is a vertical perpendicular distance between Told and the mixed reality head-mounted display; step 2: in response to the zoom-in command, obtaining a value of Znew based on the zoom-in command,wherein Znew is defined as a vertical perpendicular distance between the mixed reality head-mounted display and Tnew which is a new position of Told after being zoomed in from the original position, and then calculating Xnew-left and Xnew-right of Tnew , wherein Xnew-left is a value of XL corresponding to Tnew, and Xnew-right is a value of XR corresponding to Tnew; step 3: after obtaining Xnew-left and Xnew-right, magnifying images displayed on the left screen and the right screen by (Xnew / Xold) times, wherein Xnew represents a value of XL or XR corresponding to Tnew, and Xold represents a value of 3 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 XL or XR corresponding to Told, then translating each pixel of themagnified image on the left screen simultaneously by a same distance as how the parallax viewing position on the left screen translate when Xold-left is changed to Xnew-left when the target pixel T is zoomed in, and also translating each pixel of the magnified image on the right screen simultaneously by a same distance as how the parallax viewing position on the right screen translate whenXold-right is changed to Xnew-right when the target pixel T is zoomed in.

[0006] Preferably, the present invention also comprises step 4: in response to a zoom-out command given by the user, obtaining a new vertical distance between a further new position Tnew2 of the target pixel T and the mixed reality head- mounteddisplayafter the target pixel T is zoomedout from thenewpositionTnew in step3 to said further newpositionTnew2 in response to the zoom-out command; calculatingXnew2which is a value of X corresponding to Tnew2 using themethod of step 2; during zoom-out process, shrink themagnified image of the left screen and themagnified image of the right screen of step 3 by (Xnew2 / Xnew) times, then translating each pixel of shrunk images on the left screen and the right screen simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xnew is changed to Xnew2 when the target pixel T is zoomed out.

[0007] Preferably, step 2 comprises the following steps: given that the vertical perpendicular distanceZold betweenTold and themixed reality head-mounteddisplay is known,Xold- left is known, and Xold-right is known, a simplified formula Xnew=Xold(Zold / Znew) is used to obtain: and

[0008] Alternatively, step 2 may comprise the following steps: assuming that the target pixel T is located in a region between the left central line and the right central line, then the left screendisplays the target pixel T to a right side of the left central line, and the right screen displays the target pixel T to a left side of the right central line; in response to the zoom-in command, and based on a positional relationship between Xold-left and θold-left of Told, Xold-right and θold-right of Told, and the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display, as well as a positional relationship between Xnew-left and θnew-left of Tnew at a zoomed in position,Xnew-right andθnew-right of saidTnew, and thevertical perpendicular distanceZnewbetweensaidTnewand themixed reality head-mounted display, the following formulas are obtained: and

[0009] Since DL and DR remain unchanged when the target pixel T is zoomed in, therefore: and 4 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 thus: and

[0010] Takingpoint Land thepointRasorigins, if Told orTnew falls toa left sideof the left central lineor toa right sideof the right central line assign positive or negative values to X according to positive or negative values of the XY coordinate systems.

[0011] Preferably, if the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display determined in step 1 exceeds a preset value, preset values are assigned to Zold, Xold-left, θold-left, Xold-right, and θold-right.

[0012] Preferably, the zoom-in command or the zoom-out command are launched via assisting tools like control handles, control wristbands, and control rings, or lanuched through user’s gestures.

[0013] Ahead-mounted display device, comprising at least two cameras configured to capture target images of a target area; the head-mounted display device comprises amemory and a processor, wherein thememory is configured to store computer programs; the processor is configured to execute the computer programs to implement any aspects of the method for processing environmental image data in video see-through as described above.

[0014] A computer readable storage medium, on which a computer program is stored; the computer program, when executed by a processor, implements any aspects of the method for processing environmental image data in video see- through as described above.

[0015] According to the technical solutions of the present invention,whenazoom-in command is received fromauser, a target pixel T is determined as being the pixel closest to one end of a vertical perpendicular distance Z between the target pixel T and the mixed reality head-mounted display opposite to another end thereof at the mixed reality head-mounted display, and thencalculatingXold; anewvertical perpendicular distanceZnewbetween the target pixel Tafter beingzoomed in and the mixed reality head-mounted display is obtained based on the zoom-in command, and then calculating Xnew; next, magnifying the images on the left screen and the right screen by (Xnew / Xold) times, and then translating each pixel of the magnified images on the left screen and the right screen simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xold is changed to Xnew when the target pixel T is zoomed in. During the zoom-in process of the target pixel T, the target pixel T moves perpendicularly towards the mixed reality head-mounted display so that the vertical perpendicular distance Z is reduced while the images on the left screen and the right screen aremagnified accordingly. During the zoom-out process of the target pixel T, the target pixel Tmoves perpendicularly away from the mixed reality head-mounted display so that the vertical perpendicular distance Z is increased while the images on the left screen and the right screen are shrunk accordingly. The method for processing environment image data inVSTaccording to the present invention is implemented only aftermechanical focus adjustment of the cameras is completed. Accordingly, the present invention enables immersive viewing of the surrounding environ- ment being zoomed in or out through VST. This allows users to magnify and observe distant or very small objects, and achieves the visual perception of objects being zoomed in or out. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG.1 isa schematic illustrationof viewingasurroundingenvironment viaVSTasperceivedona left screenanda right screen, inwhich a target pixel T, a left camera and a right camera having a focal length f, and parallax between images of the left screen and the right screen are schematically illustrated. FIG. 2 is a relationship curve betweenX (XL / XR) anddistanceZby taking a logarithmof formula (2) for both the left and right sides. 5 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 FIG. 3 shows the change in parallax viewing positions during zoom in / out process of target pixel T in the present invention. FIG. 4 shows determination of the target pixel Taccording to a pixel closest to one end of a distance represented by value Z. FIG. 5 shows the relationship of the target pixel Told at the original position, Xold-left and Xold-right , θold-left and θold-right , and the vertical perpendicular distance Zold according to Embodiment 1. FIG. 6 shows the relationship of the target pixel Tnew at a zoomed-in position, Xnew-left andXnew-right , θnew-left and θnew- right , and the vertical perpendicular distance Znew according to Embodiment 1. FIG. 7 shows a functional block diagram of a head-mounted display device of the present invention. FIG. 8 illustrates the positional relationship betweenX, θ andZwhen the target pixel T is zoomed out to a newposition (i.e. target pixel Tnew2 after zoomed out) according to embodiment 1 of the present invention. FIG. 9 is a top view showing magnification or shrinking of a screen in accordance with zooming in or out operation according to embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following will clearly and thoroughly describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings. Obviously, the described embodiments are only some but not all of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtainable by skilled person in this field of art without involving invention effect shall also fall within the protection scope of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "comprising" and any variations thereof are intended to cover non-exclusive inclusion. Forexample,aprocess,method, system,product, or device that includesaseriesof stepsorunits is notnecessarily limited to those clearly listed steps or units, but may include other steps or units that are not explicitly listed or are inherent to the process, method, product, or device.

[0019] In the embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or solution described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or solutions. Rather, the use of "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0020] Technical principles of the present invention: FIG.1 is a schematic diagram showing parallax between displays shown by a left screen and a right screen corresponding toa left cameraanda right cameracapturing thesurroundingenvironment byvideosee-through (VST)wherea target pixel is indicated as T in the surrounding environment and a focal length of the left camera and the right camera is indicated as f. Here, f is particularly anoptical focal distance of the left camera and the right camera such that an imaging plane perceived byauseron the left screenand the right screen is formedat thatoptical focal distance.Assuming that the left screenand the right screen align screen pixels according to the field of view (FOV) of the left camera and the right camera respectively, a total number of pixels of an image captured by a respective camera is equal to a total number of screen pixels of a respective screen, or they are equal after correction (for example, the image captured by a camera has a total number of 2000 pixels while a respective screen has a total number of 2160 pixels; the MR glasses may choose to blacken the boundary of the respective screen so that only 2000 pixels are displayed throughVST, or scale up the image by 1.08 times (2160 / 2000). Based on the understanding herein, "after correction" refers to a proportional up / down scaling of screen pixels after the image is scaledup / down).Given that a total numberof pixels alonganX-axis of a respective screen isXtotal, and theFOVof the left cameraand the right camera is known.With reference toFIG. 1, point L is used to represent a center point of the left screenandacenter point of the left camerasimultaneously, andpointR isused to represent a centerpoint of the right screen and a center point of the right camera simultaneously, a line connecting points L and R or a line parallel to the line connecting points L andR is definedas theX-axis. EstablishXYcoordinate systemson the left screenand the right screen respectively, with points L and R being origins of the XY coordinate systems respectively. With reference to FIG. 1 again, the target pixel Twith respect to imagingplanesof the left screenand the right screenat the optical focal distance f is displayed or viewed with parallax at two different positions on the left screen and the right screen respectively (i.e. the target pixel Tappears at a right side of said point L, and the target pixel Tappears at a left side of said point R), wherein on the left screen, a distance on the imaging plane of the left screen between a point corresponding to perpendicular projection of said point L on the imaging plane of the left screen and a parallax viewing position according to which the target pixelT is viewed isdefinedasXL, andon the right screen,adistanceon the imagingplaneof the right screenbetween 6 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 a point corresponding to perpendicular projection of said point R on the imaging plane of the right screen and a parallax viewingpositionaccording towhich the target pixel T is viewed isdefinedasXR.ValuesofXLandXRare representedon the left screenandon the right screen respectively aspixel values.With reference toFIG. 1again, in the realworld,D is used to represent values of howmuch a perpendicular projection point of the target pixel Tonto the X-axis is distanced from said point L and said point R, specifically, a normal line extending from the target pixel Tand intersecting perpendicularly with the X-axis divides D into DL and DR, wherein DL is a distance between said point L and an intersection point of the normal line and the X-axis, and DR is a distance between said point R and the intersection point of the normal line and the X-axis. Define DL+DR=DIPD, wherein DIPD is an interpupillary distance between the left camera and the right camera, i.e., a fixed known distance between said point L and said point R. During zoom in / out process of the target pixel T, DL and DR remain unchanged.Asanobject as viewed from the left screenand the right screen is zoomed in, theparallax viewingpositions on the left screen and the right screen move along or parallel to the X-axis symmetrically from said point L and said point R towards a center of theMRglasses. A normal line passing perpendicularly through theX-axis at said point L is definedas a left central line, andanormal linepassingperpendicularly through theX-axisat saidpointR isdefinedasa right central line. An angle between the left central line and a line passing through said point L and the parallax viewing position on the left screen according to which the target pixel T is viewed as defined as θL; an angle between the right central line and a line passing through said point R and the parallax viewing position on the right screen according to which the target pixel T is viewed as defined as θR. Since the processes of zooming in / out the target pixel Ton the left screen and on the right screen are identical, for the sake of more convenient explanation of the principles of the present invention, formulas described below may apply to both the left screen and the right screen given that a subscript L or R is not particularly indicated.

[0021] As shown in FIG. 1, the following formulas are derived through triangular geometric calculations:

[0022] Separate calculations can be performed for the left camera and the right camera, and the following converted formula can be used in both calculations for the left camera and the right camera:

[0023] wherein Z represents a vertical perpendicular distance between the target pixel T and the MR glasses; XL represents the distance on the imaging plane of the left screen between the point corresponding to perpendicular projection of said point L on the imaging plane of the left screen and the parallax viewing position according to which the target pixel T is viewed; XR represents the distance on the imaging plane of the right screen between the point corresponding to perpendicular projection of said point R on the imaging plane of the right screen and a parallax viewing position according to which the target pixel T is viewed; f is the focal length of the left camera and the right camera; DL represents a distance in the real world between said point L representing the center point of the left screen and the intersection point of the normal line extending from the target pixel Tand intersecting perpendicularly with the X-axis; DR represents a distance in the real world between said point L representing the center point of the right screen and the intersection point of the normal line extending from the target pixel T and intersecting perpendicularly with the X-axis.

[0024] Parallax angle θ can also be used for calculation to obtain the following formulas: and since DL+DR=DIPD, then 7 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55

[0025] As shown in FIG. 3 and based on formula (2), during zoom in / out processes of the target pixel T, distance Z changes while DL and DR remain unchanged. The focal length f is a value that can only be changed during mechanical focus adjustment of the lenses of the left camera and the right camera, and because the present invention is a processing method aftermechanical focus adjustment is completed, so the focal length f also remains unchanged. Since the valueDf is constant, a simplified formula is obtained as follows:

[0026] As shown in FIG. 3, assumeTold is the target pixel Tat an original position, Xold is a distance on the imaging plane between the point corresponding to perpendicular projection of a respective point L or R on the imaging plane and a respective parallax viewing position according to which Told is viewed; Zold is a vertical perpendicular distance between Told and the MR glasses. After zooming in, Told changes to a new position, and the target pixel Tat said new position is defined as Tnew. Correspondingly, Znew representing a vertical perpendicular distance between Tnew and theMR glasses can be obtained based on user’s commands (the same value can be provided for calculations for the left screen and the right screen). From the triangular relationships ofZold andZnew,Xold andXnew,DLandDR,withTold andTnew, it canbeseen that as the target pixel T moves closer to the MR glasses (Told > Tnew), the vertical perpendicular distance between the target pixel Tand the MR glasses decreases (Zold > Znew), and the value of X increases (Xold < Xnew). Conversely, if the target pixel T moves farther away from the MR glasses, the vertical distance Z increases, and the value of X decreases.

[0027] The present invention adjusts and executes user’s zoom-in command according to the following steps: step 1: when a zoom-in command from the user is received, Zold is first obtained according to said formula (3), and a pixel closest to one end of Zold opposite another end thereof at the MR glasses is defined as Told which is the target pixel T at the original position, and accordingly, Xold is also obtained; step 2: obtaining a value of Znew based on the zoom-in command, and then calculating Xnew; step 3: magnifying images displayed on the left screen and the right screen by (Xnew / Xold) times, then translate each pixel of the magnified images on the left screen and the right screen simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xold is changed to Xnew when the target pixel T is zoomed in.

[0028] The present invention provides answers to potential problemswhichmay be raised by a person skilled in the art: Problem 1: Since Zold and Xold of the target pixel Tat the original position are not known at the time of receiving the zoom-in command, formula (4) Xnew=Xold(Zold / Znew) cannot be directly used for calculating Xnew at the time of receiving the zoom-in command. Problem2:Assume the interpupillarydistanceDIPDofbetween the left cameraand the right camera is100mm,DLand DRare both 50mm,and the focal length f of the left cameraand the right camera is 50mm.Taking the logarithmof both sides (i.e. the left camera and the right camera) of formula (2) yields a relationship curve shown in FIG. 2 between X (XL / XR) and distance Z. During zooming in / out of the target pixel T, Df is a constant number, meaning that Zmultiplied by X represents a proportional value, depending on an exact value of X, and this relationship between X and Z is visualizedasastraight lineasshown inFIG.2when logarithm is taken.When thescenesor imagescapturedby the left camera and the right camera tend to be at an infinite distance, X tends to be 0. That is to say, if the target pixel T is at an infinite position, the target pixel T will be displayed at positions on the left screen and the right screen aligning with points LandR respectively,meaning thatXL=XR=0, or θL =θR=0. This implies that during theprocessof determining Told in step (1), if Told is at an infinite position, and / or if Zold is infinite ornear to infinite, thenXold =0,making it impossible to calculate the Xnew by using formula (4).

[0029] Solution to Problem 1: Use the pixel / resolution of the left screen and the right screen and respective cameras’ field of view (FOV) as preset by the manufacturer to calculate Zold and Xold.

[0030] Thepresent inventionusesopen-sourcemodelsand tools toalign the imagesseenon the left screenand the right screen, thereby allowing comparison of each pixel. In theory, each pixel has corresponding XL and XR on the left screen and the right screen respectively.Since the left screenand the right screenof theMRglassesalignpixelsbasedon theFOV of the left camera and the right camera, a total number of pixels of an image captured by a respective camera is equal to a 8 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 total number of screen pixels of a respective screen, or they are equal after correction. Given that a total number of pixels along the X-axis of the respective screen defined as Xtotal is known, the FOV of the left camera and the right camera is known, the interpupillary distance DIPD is known, and let PPD = Xtotal / FOV, then: and byusing formula (3), Zold is calculated, andapixel closest tooneendofZold oppositeanother end thereof at theMRglasses is defined as Told which is the target pixel T at the original position. Accordingly, Xold can be obtained.

[0031] Therearemanyopen-sourceprograms that assistwithdepth calculations,which cancalculatedepth information of the images captured by the left camera and the right camera to obtain depth colors or grayscale values for each pixel in eachof the imagescapturedby the left cameraand the right camera.Alternatively, depthcolors, grayscale values, or depth information for each pixel in each of the images captured by the left camera and the right camera can be obtained using laser or time-of-flight (TOF) sensors. The existing technique for dual cameras to obtain images with depth involves: determining the internal parameters of the cameras (such as focal length, principal point position, etc.) and external parameters (position and orientation of coordinate systems relative to the real world); collecting image data of calibration boards captured by the cameras; then using MATLAB or OpenCV to calibrate the dual cameras, evaluating whether a deviationof superimposing the imagescapturedby thedual camerasmeets thepredetermined requirements, verifying the correctnessof corner point extraction, andsavingall calibration results; using thecalibration results to perform rectification of the two images captured by the same set of dual cameras, including image rectification and region of interest (ROI) cropping, to obtain row-aligned left / right views, and then performing depth estimation for the rectified views typically by usingalgorithmssuchasBM,SGBM,orGC inOpenCV toobtain parallax images; finally, an imagewith depth is calculated based on refined parallax images.

[0032] The present invention adopts existing image comparison technology to scan row by row a left image and a right imagecapturedby the left cameraand the right camera respectively, comparingall pixels ineach row tofindpixels in the left image and the right image with identical depth information or grayscale colors and consistent positional arrangements. Obtain values XL (on the left screen) and XR (on the right screen) of each of these pixels, and then calculate for each of these pixels: and as well as value Z according to said formula (3) as reinstated below:

[0033] Asshown inFIG.4, apixel closest tooneendofadistance representedbyvalueZopposite anotherend thereof at the MR glasses is defined as Told which is the target pixel Tat the original position. Save the original position data of Told, and such original position data of Told includes a value of XL corresponding to Told , specifically defined as Xold-left , a value of θL corresponding to Told, specifically defined as θold-left , a value of XR corresponding to Told , specifically definedasXold- righ, , a value of θR corresponding to Told, specifically defined as θold-right , and the value of Zold , which is a vertical 9 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 perpendicular distance between Told and the MR glasses.

[0034] After finding Told , there are two methods to calculate Xnew-left and Xnew-right during zoom-in process of Told , wherein Tnew is defined as the target pixel Tat a new position after being zoomed in, Xnew-left is defined as the value of XL corresponding to Tnew , and Xnew-right is defined as the value of XR corresponding to Tnew.

[0035] First Method: Use the simplified formula (4): Xnew=Xold(Zold / Znew): In response to the user’s zoom-in command, given that the vertical perpendicular distance Zold between Told and the MR glasses is known, Xold-left is known, and Xold-right is known, obtain Znew based on the user’s zoom-in command, then calculate: and

[0036] Second Method: Assume that the target pixel T is located in a region between the left central line and the right central line, then the left screen will display the target pixel T to a right side of the left central line, and the right screen will display the target pixel T to a left side of the right central line. As shown in FIGs. 5 and 6, in response to the user’s zoom-in command, and based on a positional relationship between Xold-left and θold-left of Told, Xold-right and θold-right of Told, and the vertical perpendicular distanceZold betweenTold and theMRglasses, aswell as a positional relationship betweenXnew-left and θnew-left of Tnew at a zoomed in position, Xnew-right and θnew-right of said Tnew, and the vertical perpendicular distance Znew between said Tnew and the MR glasses, the following formulas are obtained:

[0037] Since DL and DR remain unchanged during the zoom-in process of the target pixel T, therefore: thus: and

[0038] If Told orTnew falls toa left sideof the left central lineor toa right sideof the right central line, take thepoint Land the 10 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 point R as origins, and assign positive or negative values to X accordingly.

[0039] Under the premise that a position of the target pixel T relative to the real world as perceived by the user remains unchanged, the present invention can realize a perception that the target pixel T is zoomed in or out by using through the method described above. A traditional smartphone only has a single camera, therefore, scaling up and down of an image doesn’t create theperceptionof anobject being zoomed inandout. Sinceapair of smart glasses (MRglassesaccording to an embodiment of the present invention) has at least two cameras and a left screen and a right screen for both eyes, VST enables immersive viewing of the environment. To achieve visual perception of zooming in / out, the parallax viewing positionsof the target pixel Tbeforeandafter zooming in / out on the left screenand the right screen, aswell asa scaleof the entire imagescapturedon the left screenand the right screenbeforeandafter zooming in / outmust beadjusted.Therefore, eachpixel of themagnified (scaledup) imageson the left screenand the right screenmust be translated simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xold is changed to Xnew when the target pixel T is zoomed in / out.

[0040] Specifically, after Xnew-left and Xnew-right are obtained, magnify the entire images on the left screen and the right screen by (Xnew / Xold) times, then each pixel of the magnified (scaled up) image on the left screen must be translated simultaneously by a same distance as how the parallax viewing positions on the left screen translate when Xold-left is changed to Xnew-left, and each pixel of the magnified (scaled up) image on the right screen must be translated simultaneously by a same distance as how the parallax viewing positions on the right screen translate when Xold-right is changed to Xnew-right.

[0041] Solution toProblem2: In practice, if valueZ tends tobe infinite,ΔXafter zoom in / outwill tends to be0or otherwise very small, making users barely able to perceive the zooming effects. In this case, let the vertical perpendicular distance Zold between Told and theMR glasses as a preset definite value Nwhich represents a closer distance, for example, N can be set as 5 meters or 5000 mm; simultaneously, let the interpupillary distance DIPD between the left camera and the right camera as 100 mm, making a distance D between a point of perpendicular projection of the target pixel Tonto the X-axis and a respective point L or R being 50 mm, and the focal length f of the left camera and the right camera is 50 mm. Accordingly,Xold= (50×50) / 5000=0.5.With these twoassumedvariablesNandDIPD , use formula (4) toobtainXnewand then follow step (3) described above tomagnify images and translate the pixels to realize zoom in / out effects. Specifically, take zooming in as an example, assuming from a zoom-in command of a user that the target pixel T is zoomed in from 5 meters away, Zold = 5000 mm, Znew is obtained based on the zoom-in command, Xnew can then be obtained, and then magnify the images of the left screen and the right screen by (Xnew / Xold) times, then translate each pixel of each of the magnified images on the left screen and the right screen simultaneously by a same distance towards an interior of the respective image as how the parallax viewing positions translate when Xold is changed to Xnew. Embodiment 1

[0042] The first embodiment of the present invention provides a method for processing environmental image data in video see-through (VST), applicable to a systemof amixed reality (MR)head-mounteddisplay; at least two camerasof the system for two eyes of a user capture real environment images, which are displayed on a left screen and a right screen of the head-mounted display for users to view. Based on user’s interactive commands, the system post-processes the real environment images to create visual effects of zooming in / out of objects within the real environment images. The interactive commands include preset zoom-in and zoom-out commands. The system predicts a zoom distance based on the interactive commands. The real environment images refer to images of the real world captured by said at least two cameras. The system post-processes the real environment images according to the following steps:

[0043] step 1: in response to a zoom-in command given by the user who views the real environment images through VST, determining a target pixel T based on a distance Zwhich is a vertical perpendicular distance between the target pixel Tand themixed reality (MR) head-mounted display, wherein a pixel closest to one end of said distance Z opposite another end thereof at themixed reality (MR) head-mounted display is defined as the target pixel T; step 1 comprises the following steps:

[0044] assuming the left screen and the right screen of themixed reality (MR) head-mounted display align pixels based on field of view (FOV) of a left camera and a right camera of the system respectively, a total number of pixels of an image captured by a respective one of the left camera and right camera is equal to a total number of screen pixels of a respective one of the left screen and the right screen either with or without correction; given that a total number of pixels along an X- axis of a respective one of the left screen and the right screen is Xtotal, the FOVof the left camera and the right camera is known, an interpupillary distance DIPD between the left camera and the right camera is known, and let a pixel per degree (PPD) being PPD = Xtotal / FOV; using point L to represent a center point of the left screen and a center point of the left camera simultaneously, and using point R to represent a center point of the right screen and a center point of the right camera simultaneously, a line connecting said point L and said point R or a line parallel to the line connecting said point L and said point R is defined as the X-axis; establishing XY coordinate systems on the left screen and the right screen respectively, with said point L and said point R being origins of the XY coordinate systems respectively; wherein on the left 11 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 screen, a distanceonan imagingplaneof the left screenbetweenapoint corresponding to perpendicular projectionof said point L on the imagingplaneof the left screenandaparallax viewingposition according towhich the target pixel T is viewed is definedasXL, and on the right screen, a distance on an imaging plane of the right screen between apoint corresponding to perpendicular projection of said point R on the imaging plane of the right screen and a parallax viewing position according to which the target pixel T is viewed is defined as XR; values of said XL and said XR are represented on the left screenandon the right screen respectively as pixel values; as anobject as viewed from the left screenand the right screen is zoomed in, the parallax viewing positions on the left screen and the right screen move along or parallel to the X-axis symmetrically fromsaidpoint LandsaidpointR towardsacenterof themixed reality (MR)head-mounteddisplay; anormal line passing perpendicularly through the X-axis at said point L is defined as a left central line, and a normal line passing perpendicularly through theX-axis at said pointR is definedasa right central line; anangle between the left central lineand a linepassing throughsaidpoint Land theparallaxviewingpositionon the left screenaccording towhich the targetpixelT is viewed as defined as θL; an angle between the right central line and a line passing through said point R and the parallax viewing position on the right screen according to which the target pixel T is viewed as defined as θR; in the real world, D is used to represent valuesof howmuchaperpendicular projectionpoint of the target pixel Tonto theX-axis is distanced from said point L and said point R, wherein a normal line extending from the target pixel Tand intersecting perpendicularly with the X-axis divides D into DL and DR, wherein DL is a distance between said point L and an intersection point of the X-axis and the normal line extending from the target pixel Tand intersecting perpendicularly with the X-axis, and DR is a distance between said point R and the intersection point of the X-axis and the normal line extending from the target pixel T and intersecting perpendicularlywith theX-axis; definingDL+DR=DIPD,whereinDIPD is the interpupillary distancebetween the left camera and the right camera andalso a fixed knowndistancebetween said point L and said pointR; during zoom in / out process of the target pixel T, DL and DR remain unchanged;

[0045] scanning rowby rowa left image anda right image captured by the left camera and the right camera respectively, comparing all pixels in each row to find pixels in the left image and the right image with identical depth information or grayscale colors and consistent positional arrangements; Obtaining values of XL and XR of each of these pixels, and then calculate for each of these pixels: and as well as the distance Z according to the following formula: defining the pixel closest to said one end of said distance Z opposite said another end thereof at the mixed reality (MR) head-mounted display as the target pixel T, which is currently at an original position; said target pixel T at said original position is defined as Told; saving the original position data of Told, and said original position data of Told includes a value of XL corresponding to Told, defined as Xold-left , a value of θL corresponding to Told, defined as θold-left , a value of XR corresponding to Told , defined as Xold-right , a value of θR corresponding to Told, defined as θold-right , and a value of Zold , which is a vertical perpendicular distance between Told and the mixed reality (MR) head-mounted display.

[0046] Step 2: in response to the user’s zoom-in command, obtaining a value of Znew based on the zoom-in command, wherein Znew is defined as a vertical perpendicular distance between the mixed reality (MR) head-mounted display and Tnewwhich is the target pixel Tat a newposition after being zoomed in from the original position, and then calculatingXnew- left and Xnew-right of Tnew as follows, wherein Xnew-left is a value of XL corresponding to Tnew, and Xnew-right is a value of XR corresponding to Tnew:

[0047] First Method: given that the vertical perpendicular distance Zold between Told and the mixed reality (MR) head- mounted display is known, Xold-left is known, and Xold-right is known, a simplified formula Xnew=Xold(Zold / Znew) is used to obtain: 12 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 and

[0048] Second Method: assuming that the target pixel T is located in a region between the left central line and the right central line, then the left screen displays the target pixel T to a right side of the left central line, and the right screen displays the target pixelT toa left sideof the right central line; in response to theuser’szoom-in command,andbasedonapositional relationship between Xold-left and θold-left of Told, Xold-right and θold-right of Told, and the vertical perpendicular distance Zold between Told and the mixed reality (MR) head-mounted display, as well as a positional relationship between Xnew-left and θnew-left of Tnew at a zoomed in position, Xnew-right and θnew-right of said Tnew, and a vertical perpendicular distance Znew between said Tnew and the mixed reality (MR) head-mounted display, the following formulas are obtained:

[0049] Since DL and DR remain unchanged when the target pixel T is zoomed in, therefore: thus: and

[0050] As shown in FIGs. 5 and 6, since point L and the point R are taken as origins, if Told or Tnew falls to a left side of the left central lineor toa right sideof the right central line, simplyassignpositiveornegativevalues toXaccording topositiveor negative values of the XY coordinate systems.

[0051] It should be noted that the calculated X values may exceed screen dimensions. Each mixed reality (MR) head- mounteddisplay (e.g.MRglasses)hasaspecificFOV.Forexample, inascreenhaving100°FOVand2000X-axispixels, a left side and a right side of the screen each being of 50° FOV will each contain 1000 pixels or a number of pixels being (Xtotal / 2), given that a center of the screen is taken as the origin. If X exceeds ±1000 pixels, it cannot be displayed.

[0052] Step 3: As shown in FIG. 9, after obtainingXnew-left andXnew-right, magnifying images displayed on the left screen and the right screen by (Xnew / Xold) times, then translating each pixel of the magnified image on the left screen 13 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 simultaneously by a same distance as how the parallax viewing position on the left screen translate when Xold-left is changed toXnew-left when the target pixel T is zoomed in, and also translating eachpixel of themagnified imageon the right screen simultaneously by a samedistanceashow theparallax viewing position on the right screen translatewhenXold-right is changed to Xnew-right when the target pixel T is zoomed in.

[0053] Asshown inFIGs.5and6,whenTold is zoomed in to thenewposition,Xold-left andXold-right areadjusted toXnew-left andXnew-right. Therefore, pixel positions of the real world imagesmust also be proportionally adjusted. Let Xtotal-old = Xtotal, thenXtotal-new =Xtotal-old * (Xold-left / Xnew-left) = Xtotal * ( Xold-left / Xnew-left), andPPDnew =PPDold * (Xold-left / Xnew-left). Since the number of total pixels of the screen remains Xtotal, each pixel of the real world image displayed on each pixel of the screen before imagemagnification cannowonly bedisplayedbyevery (Xnew / Xold) pixels of the screenafter imagemagnification. For example, if the total number of screen pixels and a total number of pixels of a real world image is consistent, for example, having 2000 total pixels, and given 100° FOVand PPDold being 20, and given that Xtotal-new is now 1000, then PPDnew is 10. Given that the total number of screen pixels remains 2000, it can be concluded that each pixel of the real world image displayed on each pixel of the screen before imagemagnification cannowonly be displayed by every 2 pixels of the screen after image magnification. In other words, the image is magnified by two times.

[0054] Step 4: As shown in FIG. 8, in response to a user’s zoom-out command, obtaining a new vertical distance between a further newposition Tnew2 of the target pixel Tand themixed reality (MR) head-mounted display after the target pixel T is zoomedout from thenewpositionTnew in step 3 to said further newpositionTnew2 in response to theuser’s zoom- out command; calculating Xnew2which is a value of X corresponding to Tnew2 using themethod of step 2. During zoom-out process, as shown in FIG. 9, shrink the magnified image of the left screen and the magnified image of the right screen of step3by (Xnew2 / Xnew) times, then translatingeachpixel of shrunk imageson the left screenand the right screenbyasame distance as how the parallax viewing positions on the left screen and the right screen translate when Xnew is changed to Xnew2 when the target pixel T is zoomed out.

[0055] Topreventblackboundarieson the left screenand the right screenduring thezoom-out processwhen the images are shrunk into excessively small size, limit the zoom-out process to stop when image size returns to its original size captured by the cameras. Alternatively, continued zoom-out process may be allowed, but this will result in the captured images smaller than the respective screens, resulting in shrunk images surrounded by black color or AI-generated virtual environment constructed through AIGC technology.

[0056] If the vertical perpendicular distance Zold between Told and the mixed reality (MR) head-mounted display determined in step 1 exceeds a preset value, preset values are assigned to Zold, Xold-left, θold-left, Xold-right, and θold-right.

[0057] The user’s zoom-in / out commands can be launched via assisting tools like control handles, control wristbands and control rings, or they can be launched through user’s gestures. User’s zoom-in / out commands and how they can be launched can be customized and are already known in the prior arts. The present invention is not intended to provide technical solutions to these issues, so therefore they will not be discussed in the present invention.

[0058] The present invention predicts the zoom-in / out distance based on interactive commands, which can be implemented through various existing techniques - for example, by converting the duration specified in the interactive commands or by deriving from amplitude of the user’s gestures. These issues do not constitute part of the inventive technical solutions of the present invention, so therefore they will not be discussed herein.

[0059] A person skilled in this field of art should further realize that, the units and algorithm steps of the examples described with reference to the disclosed embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functions in the foregoing description.Whether these functions are implemented by hardware or software depends on the specific applications and design constraints of the technical solutions. A person skilled in this field of art can use differentmethods to implement the described functions for each specific application, but all those implementations should not be considered exceeding the scope of the present invention.

[0060] Specifically, the steps of themethod embodiments of the present invention can be completed by integrated logic circuits in a hardware of a processor and / or by software instructions from the processor. The steps of the methods disclosed with reference to the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the hardware decoding processor. The softwaremodules can be located in known storagemedium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the method embodiments in combination with its hardware. Embodiment 2

[0061] Thesecondembodiment of thepresent inventionprovidesahead-mounteddisplay device, as shown inFigure 7. The head-mounted display device 700 comprises: a memory 710 and a processor 720, wherein the memory 710 is 14 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 configured to store computer programs and transmit program codes to the processor 720. In other words, the processor 720 can call and execute the computer programs from the memory 710 to implement the method in embodiment 1 of the present invention. For example, the processor 720 can execute the method described in embodiment 1 according to the instructions stored in the computer programs.

[0062] In some embodiments of the present invention, the processor 720 includes but not limited to:

[0063] ageneral-purposeprocessor, adigital signal processor (DSP), anapplication-specific integratedcircuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.

[0064] In some embodiments of the present invention, the memory 710 includes but not limited to: volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as external cachememory.Bywayof non-limiting examples,many formsofRAMareavailable, suchas static randomaccess memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), doubledata ratesynchronousdynamic randomaccessmemory (DDRSDRAM),enhancedsynchronousdynamic random accessmemory (ESDRAM), synch link dynamic randomaccessmemory (SLDRAM), and directRambusRAM (DRRAM).

[0065] In some embodiments of the present invention, a computer programmay be divided into one or more modules, which are stored in thememory 710 and executed by the processor 720 to complete themethod described in embodiment 1of thepresent invention. Theoneormoremodulesmaybeaseries of computer program instruction segments capable of performing specific functions, said computer program instruction segments are configured to describe execution processes of the computer program in the head-mounted display device 700.

[0066] As shown in FIG. 7, the head-mounted display device 700 also comprises: a transceiver 730 connected to the processor 720 or thememory 710. Theprocessor 720may control the transceiver 730 to communicatewith other devices, specifically to send information or data to other devices or receive information or data sent by the other devices. The transceiver 730 may comprise at least two cameras for capturing target images of a target area.

[0067] It shouldbeunderstood that various components in thehead-mounteddisplaydevice700are connected through a bus system. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. Embodiment 3:

[0068] The third embodiment of the present invention also provides a computer storage medium on which a computer program is stored; the computer program, when executed by a computer, enables the computer to execute the method described in embodiment 1 of the present invention.

[0069] The specific implementations described above explain the objectives, technical solutions, and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific implementations of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, or improvementsmadewithin the spirit and principles of the present invention shall be included in the protection scope of the present invention. Claims 1. A method for processing environmental image data in video see-through, applicable to a system of a mixed reality head-mounted display; at least two cameras of the system corresponding to two eyes of a user capture real environment images, which are displayed and viewed on a left screen and a right screen of the mixed reality head-mounted display; wherein: based on user’s interactive commands, the system post-processes the real environment images to create visual effects of zooming in / out of objects within the real environment images; the interactive commands include preset zoom-in and zoom-out commands; the system predicts a zoom distance based on the interactive commands; the system post-processes the real environment images according to the following steps: step1: in response toazoom-in commandgivenby theuserwhoviews the real environment images throughVST, determining a target pixel T based on a distance Z which is a vertical perpendicular distance between the target pixel Tand themixed reality head-mounted display, wherein a pixel closest to one end of said distanceZ opposite another end thereof at the mixed reality head-mounted display is defined as the target pixel T; step 1 comprises the following steps: assuming the left screenand the right screenof themixed reality head-mounteddisplay alignpixels basedon 15 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 fieldof view (FOV)of a left cameraanda right cameraof thesystem respectively, a total numberof pixelsof an image captured by a respective one of the left camera and right camera is equal to a total number of screen pixels of a respective one of the left screen and the right screen either with or without correction; given that a total number of pixels along an X-axis of a respective one of the left screen and the right screen is Xtotal, the FOV of the left camera and the right camera is known, an interpupillary distance (DIPD) between the left cameraand the right camera is known,and let apixel perdegree (PPD)beingPPD=Xtotal / FOV;usingpoint L to represent a center point of the left screen and a center point of the left camera simultaneously, and using point R to represent a center point of the right screen and a center point of the right camera simultaneously, a line connecting said point L and said point R or a line parallel to the line connecting said point L and said point R is defined as the X-axis; establishing XY coordinate systems on the left screen and the right screen respectively, with said point L and said point R being origins of the XY coordinate systems respectively; wherein on the left screen, a distanceonan imagingplaneof the left screenbetweenapoint corresponding to perpendicular projection of said point L on the imaging plane of the left screen anda parallax viewing position according to which the target pixel T is viewed is defined as XL, and on the right screen, a distance on an imagingplaneof the right screenbetweenapoint corresponding toperpendicular projectionof saidpointRon the imaging plane of the right screen and a parallax viewing position according to which the target pixel T is viewed is defined as XR; values of said XL and said XR are represented on the left screen and on the right screen respectivelyaspixel values;asanobject asviewed from the left screenand the right screen is zoomed in, the parallax viewing positions on the left screen and the right screen move along or parallel to the X-axis symmetrically from said point L and said point R towards a center of themixed reality head-mounted display; a normal line passing perpendicularly through the X-axis at said point L is defined as a left central line, and a normal line passing perpendicularly through the X-axis at said point R is defined as a right central line; an angle between the left central line anda line passing through said point L and the parallax viewing position on the left screen according to which the target pixel T is viewed as defined as θL; an angle between the right central line and a line passing through said point R and the parallax viewing position on the right screen according towhich the target pixel T is viewedasdefinedasθR; in the realworld,D is used to represent values of howmuchaperpendicular projectionpoint of the target pixel Tonto theX-axis is distanced fromsaidpoint L and said pointR,wherein a normal line extending from the target pixel Tand intersecting perpendicularlywith theX-axis dividesD intoDLandDR,whereinDL is a distancebetween said point L and an intersection point of the X-axis and the normal line extending from the target pixel Tand intersecting perpendicularly with the X- axis, and DR is a distance between said point R and the intersection point of the X-axis and the normal line extending from the target pixel T and intersecting perpendicularly with the X-axis; defining DL+DR=DIPD, wherein DIPD is the interpupillary distance between the left camera and the right camera and also a fixed knowndistancebetweensaid point L andsaidpointR; during zoom in / out processof the target pixel T,DLand DR remain unchanged; scanning row by row a left image and a right image captured by the left camera and the right camera respectively, comparing all pixels in each row to find pixels in the left image and the right image with identical depth information or grayscale colors and consistent positional arrangements; obtaining values of XL andXR of each of these pixels, and then calculate for each of these pixels: and as well as the distance Z according to the following formula: defining the pixel closest to said one end of said distance Z opposite said another end thereof at the mixed reality head-mounted display as the target pixel T, which is currently at an original position; said target pixel T at said original position is defined as Told; saving original position data of Told, wherein said original position 16 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 data of Told includes a value of XL corresponding to Told , defined as Xold-left , a value of θL corresponding to Told, definedasθbold-left , a valueofXRcorresponding toTold , definedasXold-right , a valueofθRcorresponding toTold, definedasθold-right , andavalueofZold ,which isavertical perpendiculardistancebetweenToldand the mixed reality head-mounted display; step 2: in response to the zoom-in command, obtaining a value of Znew based on the zoom-in command, wherein Znew is defined as a vertical perpendicular distance between the mixed reality head-mounted display and Tnew which is a new position of Told after being zoomed in from the original position, and then calculating Xnew-left and Xnew-right of Tnew , wherein Xnew-left is a value of XL corresponding to Tnew, and Xnew-right is a value of XR corresponding to Tnew; step 3: after obtaining Xnew-left and Xnew-right, magnifying images displayed on the left screen and the right screen by (Xnew / Xold) times, wherein Xnew represents a value of XL or XR corresponding to Tnew, and Xold represents a value of XL or XR corresponding to Told, then translating each pixel of the magnified image on the left screen simultaneously by a same distance as how the parallax viewing position on the left screen translate when Xold-left is changed to Xnew-left when the target pixel T is zoomed in, and also translating each pixel of themagnified image on the right screen simultaneously by a same distance as how the parallax viewing position on the right screen translate when Xold-right is changed to Xnew-right when the target pixel T is zoomed in. 2. The method for processing environmental image data in video see-through of claim 1, further comprising step 4: in response to a zoom-out command given by the user, obtaining a new vertical distance between a further newposition Tnew2 of the target pixel Tand the mixed reality head-mounted display after the target pixel T is zoomed out from the newpositionTnew in step 3 to said further newpositionTnew2 in response to the zoom-out command; calculatingXnew2 which isavalueofXcorresponding toTnew2using themethodof step2; duringzoom-out process, shrink themagnified image of the left screen and themagnified image of the right screen of step 3 by (Xnew2 / Xnew) times, then translating each pixel of shrunk images on the left screen and the right screen simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xnew is changed to Xnew2 when the target pixel T is zoomed out. 3. The method for processing environmental image data in video see-through of claim 1, wherein step 2 comprises the following steps: given that the vertical perpendicular distance Zold between Told and themixed reality head-mounted display is known, Xold-left is known, and Xold-right is known, a simplified formula Xnew=Xold(Zold / Znew) is used to obtain: and 4. The method for processing environmental image data in video see-through of claim 1, wherein step 2 comprises the following steps: assuming that the target pixel T is located in a region between the left central line and the right central line, then the left screendisplays the target pixelT toa right sideof the left central line, and the right screendisplays the target pixel T toa left side of the right central line; in response to the zoom-in command, and based on a positional relationship between Xold-left and θold-left of Told, Xold-right and θold-right of Told, and the vertical perpendicular distance Zold between Told and themixed reality head-mounted display, as well as a positional relationship between Xnew-left and θnew-left of Tnew at a zoomed in position, Xnew-right and θnew-right of said Tnew, and the vertical perpendicular distance Znew between said Tnew and the mixed reality head-mounted display, the following formulas are obtained: and 17 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 since DL and DR remain unchanged when the target pixel T is zoomed in, therefore: and thus: and taking the point L and the point R as origins, if Told or Tnew falls to a left side of the left central line or to a right side of the right central line, assign positive or negative values to X according to positive or negative values of the XY coordinate systems. 5. The method for processing environmental image data in video see-through of claim 1, wherein if the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display determined in step 1 exceeds a preset value, preset values are assigned to Zold, Xold-left, θold-left, Xold-right, and θold-right. 6. Ahead-mounteddisplay device (700), comprisingat least twocameras configured to capture target imagesof a target area; the head-mounted display device comprises amemory (710) and a processor (720), wherein thememory (710) is configured to store computer programs; the processor (720) is configured to execute the computer programs to implement the method for processing environmental image data in video see-through as in any one of claims 1 to 5. 7. Acomputer readablestoragemedium,onwhichacomputerprogram isstored; thecomputer program,whenexecuted byaprocessor (720), implements themethod for processing environmental imagedata in video see-throughas in any one of claims 1 to 5. 18 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 19 EP 4 730 765 A1 20 EP 4 730 765 A1 21 EP 4 730 765 A1 22 EP 4 730 765 A1 23 EP 4 730 765 A1 24 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 25 EP 4 730 765 A1 5 10 15 20 25 30 35 40 45 50 55 摘要 处理环境图像数据的方法及其相关产品，其应用于扩展现实头显设备，其 具有最少两个采集真实环境图像的摄像头，真实环境图像通过屏幕向用户展 示。响应于用户的拉近指令，基于目标像素 T 与混合现实头显设备之间的垂直 距离 Z 确定目标像素 T，然后计算 Xold；在目标像素 T 拉近后得到新的垂直距 离 Znew，然后计算 Xnew；把屏幕的画面放大（Xnew / Xold）倍，然后将屏幕放大 画面的每个像素都平移一段距离，该距离与目标像素 T 拉近时当 Xold 变成 Xnew 时屏幕的视差位置平移的距离相同。

Claims

1. A method for processing environmental image data in video see-through, applicable to a system of a mixed reality head-mounted display; at least two cameras of the system corresponding to two eyes of a user capture real environment images, which are displayed and viewed on a left screen and a right screen of the mixed reality head-mounted display; wherein: based on user's interactive commands, the system post-processes the real environment images to create visual effects of zooming in / out of objects within the real environment images; the interactive commands include preset zoom-in and zoom-out commands; the system predicts a zoom distance based on the interactive commands; the system post-processes the real environment images according to the following steps: step 1: in response to a zoom-in command given by the user who views the real environment images through VST, determining a target pixel T based on a distance Z which is a vertical perpendicular distance between the target pixel T and the mixed reality head-mounted display, wherein a pixel closest to one end of said distance Z opposite another end thereof at the mixed reality head-mounted display is defined as the target pixel T; step 1 comprises the following steps: assuming the left screen and the right screen of the mixed reality head-mounted display align pixels based on field of view (FOV) of a left camera and a right camera of the system respectively, a total number of pixels of an image captured by a respective one of the left camera and right camera is equal to a total number of screen pixels of a respective one of the left screen and the right screen either with or without correction; given that a total number of pixels along an X-axis of a respective one of the left screen and the right screen is Xtotal, the FOV of the left camera and the right camera is known, an interpupillary distance (DIPD) between the left camera and the right camera is known, and let a pixel per degree (PPD) being PPD = Xtotal / FOV; using point L to represent a center point of the left screen and a center point of the left camera simultaneously, and using point R to represent a center point of the right screen and a center point of the right camera simultaneously, a line connecting said point L and said point R or a line parallel to the line connecting said point L and said point R is defined as the X-axis; establishing XY coordinate systems on the left screen and the right screen respectively, with said point L and said point R being origins of the XY coordinate systems respectively; wherein on the left screen, a distance on an imaging plane of the left screen between a point corresponding to perpendicular projection of said point L on the imaging plane of the left screen and a parallax viewing position according to which the target pixel T is viewed is defined as XL, and on the right screen, a distance on an imaging plane of the right screen between a point corresponding to perpendicular projection of said point R on the imaging plane of the right screen and a parallax viewing position according to which the target pixel T is viewed is defined as XR; values of said XL and said XR are represented on the left screen and on the right screen respectively as pixel values; as an object as viewed from the left screen and the right screen is zoomed in, the parallax viewing positions on the left screen and the right screen move along or parallel to the X-axis symmetrically from said point L and said point R towards a center of the mixed reality head-mounted display; a normal line passing perpendicularly through the X-axis at said point L is defined as a left central line, and a normal line passing perpendicularly through the X-axis at said point R is defined as a right central line; an angle between the left central line and a line passing through said point L and the parallax viewing position on the left screen according to which the target pixel T is viewed as defined as θL; an angle between the right central line and a line passing through said point R and the parallax viewing position on the right screen according to which the target pixel T is viewed as defined as θR; in the real world, D is used to represent values of how much a perpendicular projection point of the target pixel T onto the X-axis is distanced from said point L and said point R, wherein a normal line extending from the target pixel T and intersecting perpendicularly with the X-axis divides D into DL and DR, wherein DL is a distance between said point L and an intersection point of the X-axis and the normal line extending from the target pixel T and intersecting perpendicularly with the X-axis, and DR is a distance between said point R and the intersection point of the X-axis and the normal line extending from the target pixel T and intersecting perpendicularly with the X-axis; defining DL+DR=DIPD, wherein DIPD is the interpupillary distance between the left camera and the right camera and also a fixed known distance between said point L and said point R; during zoom in / out process of the target pixel T, DL and DR remain unchanged; scanning row by row a left image and a right image captured by the left camera and the right camera respectively, comparing all pixels in each row to find pixels in the left image and the right image with identical depth information or grayscale colors and consistent positional arrangements; obtaining values of XL and XR of each of these pixels, and then calculate for each of these pixels: θ L = X L / PPD , and θ R = X R / PPD , as well as the distance Z according to the following formula: Z = D IPD TAN θ L + TAN θ R ; defining the pixel closest to said one end of said distance Z opposite said another end thereof at the mixed reality head-mounted display as the target pixel T, which is currently at an original position; said target pixel T at said original position is defined as Told; saving original position data of Told, wherein said original position data of Told includes a value of XL corresponding to Told , defined as Xold-left , a value of θL corresponding to Told, defined as θbold-left , a value of XR corresponding to Told , defined as Xold-right , a value of θR corresponding to Told, defined as θold-right , and a value of Zold , which is a vertical perpendicular distance between Told and the mixed reality head-mounted display; step 2: in response to the zoom-in command, obtaining a value of Znew based on the zoom-in command, wherein Znew is defined as a vertical perpendicular distance between the mixed reality head-mounted display and Tnew which is a new position of Told after being zoomed in from the original position, and then calculating Xnew-left and Xnew-right of Tnew , wherein Xnew-left is a value of XL corresponding to Tnew, and Xnew-right is a value of XR corresponding to Tnew; step 3: after obtaining Xnew-left and Xnew-right, magnifying images displayed on the left screen and the right screen by (Xnew / Xold) times, wherein Xnew represents a value of XL or XR corresponding to Tnew, and Xold represents a value of XL or XR corresponding to Told, then translating each pixel of the magnified image on the left screen simultaneously by a same distance as how the parallax viewing position on the left screen translate when Xold-left is changed to Xnew-left when the target pixel T is zoomed in, and also translating each pixel of the magnified image on the right screen simultaneously by a same distance as how the parallax viewing position on the right screen translate when Xold-right is changed to Xnew-right when the target pixel T is zoomed in.

2. The method for processing environmental image data in video see-through of claim 1, further comprising step 4: in response to a zoom-out command given by the user, obtaining a new vertical distance between a further new position Tnew2 of the target pixel T and the mixed reality head-mounted display after the target pixel T is zoomed out from the new position Tnew in step 3 to said further new position Tnew2 in response to the zoom-out command; calculating Xnew2 which is a value of X corresponding to Tnew2 using the method of step 2; during zoom-out process, shrink the magnified image of the left screen and the magnified image of the right screen of step 3 by (Xnew2 / Xnew) times, then translating each pixel of shrunk images on the left screen and the right screen simultaneously by a same distance as how the parallax viewing positions on the left screen and the right screen translate when Xnew is changed to Xnew2 when the target pixel T is zoomed out.

3. The method for processing environmental image data in video see-through of claim 1, wherein step 2 comprises the following steps: given that the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display is known, Xold-left is known, and Xold-right is known, a simplified formula Xnew=Xold(Zold / Znew) is used to obtain: X new − left = X old − left Z old / Z new ,and X new − right = X old − right Z old / Z new .

4. The method for processing environmental image data in video see-through of claim 1, wherein step 2 comprises the following steps: assuming that the target pixel T is located in a region between the left central line and the right central line, then the left screen displays the target pixel T to a right side of the left central line, and the right screen displays the target pixel T to a left side of the right central line; in response to the zoom-in command, and based on a positional relationship between Xold-left and θold-left of Told, Xold-right and θold-right of Told, and the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display, as well as a positional relationship between Xnew-left and θnew-left of Tnew at a zoomed in position, Xnew-right and θnew-right of said Tnew, and the vertical perpendicular distance Znew between said Tnew and the mixed reality head-mounted display, the following formulas are obtained: D R = Z old ∗ TAN θ old − right , and D L = Z old ∗ TAN θ old − left ; since DL and DR remain unchanged when the target pixel T is zoomed in, therefore: θ new − left = TAN − 1 D L Z new , and θ new − right = TAN − 1 D R Z new ; thus: X new − left = θ new − left * PPD , and X new − right = θ new − right * PPD ; taking the point L and the point R as origins, if Told or Tnew falls to a left side of the left central line or to a right side of the right central line, assign positive or negative values to X according to positive or negative values of the XY coordinate systems.

5. The method for processing environmental image data in video see-through of claim 1, wherein if the vertical perpendicular distance Zold between Told and the mixed reality head-mounted display determined in step 1 exceeds a preset value, preset values are assigned to Zold, Xold-left, θold-left, Xold-right, and θold-right.

6. A head-mounted display device (700), comprising at least two cameras configured to capture target images of a target area; the head-mounted display device comprises a memory (710) and a processor (720), wherein the memory (710) is configured to store computer programs; the processor (720) is configured to execute the computer programs to implement the method for processing environmental image data in video see-through as in any one of claims 1 to 5.

7. A computer readable storage medium, on which a computer program is stored; the computer program, when executed by a processor (720), implements the method for processing environmental image data in video see-through as in any one of claims 1 to 5.

Images