Processing method of environmental image data in camera perspective VST, head display device and storage medium

Abstract

The present invention discloses a method for processing environmental image data in a camera perspective VST, a head display device and a storage medium, which belongs to the field of camera perspective technology and is applied to a system of an XR extended reality head-mounted display device. A binocular camera collects real environmental images, which are displayed on the screen of the head-mounted display device for users to watch. In response to a user's zoom-in instruction, a pixel closest to the X-axis of the glasses is first found as a target pixel, and a parallax position X of the target pixel at the original position is obtained. old ; Calculate the parallax position X at the new position new ; First, enlarge the entire screen on the left and right (X new / X old ) times, and then enlarge each pixel of the left and right screens to follow the target pixel from the parallax position X old Pan to parallax position X new The present invention performs post-processing on the environmental image data in the camera perspective VST after the camera is mechanically focused, so that the user can immersively see the visual effect of the surrounding environment being brought closer through the VST method, and can zoom in to view distant or too small objects.

Description

Technical Field

[0001] The present invention relates to the field of camera perspective technology, and in particular to a method for processing environmental image data in a camera perspective VST, a head display device and a storage medium. Background Art

[0002] XR glasses are divided into two categories: AR augmented reality and VR virtual reality. AR glasses use Optical SeeThrough or optical lenses to achieve perspective and see the surrounding environment (hereinafter referred to as optical perspective OST). VR glasses are purely virtual glasses that cannot see the outside environment. In recent years, MR mixed reality glasses have begun to appear, that is, VR uses a camera to view the surrounding environment. This is achieved by using Video See Through or Visual Pass Through or video perspective (hereinafter referred to as video perspective VST). Since the optical perspective OST does not capture images or video streams of the environment, it is impossible to process the video or image of the environment. On the contrary, since the video perspective VST uses a camera to capture images or video streams of the surrounding environment, the surrounding environment can be calculated and post-processed. In addition to rendering and superimposing customized virtual objects and backgrounds as display content, the Metaverse also needs to be able to post-process the surrounding environment.

[0003] On traditional tablets and mobile phones, you can use two fingers to zoom in and out the image or video stream you see. Especially when shooting videos, there are already functions to zoom in and out (Zoom In / Zoom Out) or change the focus distance f. With the advancement of technology, MR (mixed reality) glasses will gradually become an extension of our human eyes, and we need to use glasses to zoom in and out to see distant or too small objects. The zoom in and out function is easy to implement on a two-dimensional screen because there is no need to consider the problem of parallax, that is, zoom in and out is equivalent to zooming in and out. But in binocular MR glasses, zooming in and out are different events from zooming in and out. Zooming in and out is a simple thing, but zooming in and out requires adjusting the parallax of the left and right eyes. In traditional MR glasses, there is currently no method to perform zoom in and out post-processing on the environmental image data in the camera perspective VST. Summary of the invention

[0004] The purpose of the present invention is to provide a method for processing environmental image data in a camera perspective VST, a head display device and a storage medium, which are applied to a system of an XR extended reality head-mounted display device. Based on the user's interactive instructions, the environmental image data in the camera perspective VST is post-processed, so that the user can immersively see the surrounding environment being zoomed in or pushed away through the VST, can zoom in to view distant or too small objects, and achieve the visual effect of objects being zoomed in or pushed away.

[0005] The method for processing environmental image data in a camera perspective VST disclosed in the present invention is applied to a system of an XR extended reality head-mounted display device. A binocular camera collects a real environment image, which is displayed on a screen of the head-mounted display device for viewing by a user. Based on the user's interactive instructions, the system performs post-processing on the real environment image so that objects in the real environment image have a visual effect of being zoomed in or out. The interactive instructions include preset zoom-in instructions and zoom-out instructions. The system predicts the zoom-in or zoom-out distance according to the interactive instructions. The post-processing includes the following steps:

[0006] Step 1: In response to a zoom-in instruction given by the user when viewing the real environment image through the VST method, start looking for the pixel closest to the glasses as the target pixel T:

[0007] Assume that the left and right screens of the glasses are aligned according to the field of view FOV of the left and right cameras, that is, the total number of pixels of the camera image is the same as the total number of pixels of the screen or the same after correction. total , the field of view FOV of the left and right cameras, the pupil distance D of the left and right cameras IPD and pixels per degree ; Point L and point R are used to represent the center points of the left and right screens and the center points of the left and right cameras. The line connecting point L and point R or the line parallel to them is set as the X axis. An XY coordinate system with point L and point R as the origin is established on the left and right screens respectively. The distance between the parallax position of the target pixel T displayed on the left screen and point L is X L The right screen shows the distance between the disparity position of the target pixel T and point R, which is X R , the distance value X L and X R It is expressed in pixel values ​​of the left and right screens. When the objects on the left and right screens move from far to near, the parallax positions are adjusted symmetrically from point L and point R along the X-axis toward the middle of the glasses. The normal of point L is called the left center line, and the normal of point R is called the right center line. The parallax angle and They represent the degrees of the parallax position when the left / right center line is deviated toward the center of the glasses respectively; in the real world, D represents the actual distance between the target pixel T and point L or point R on the X-axis, and the distance between the target pixel T and the normal of the glasses cutting the X-axis is D L and right distance D R , where D L +D R =D IPD , the D IPD is the pupil distance of the left and right cameras, that is, the distance between point L and point R, which is a known fixed value. When the target pixel T is zoomed in and out, D L and D R Fixed and unchanging;

[0008] Scan the left and right views captured by the camera line by line, compare all the pixels in each line, find the pixels with the same depth information or grayscale color and the same arrangement in the left and right views, and obtain the parallax position X of these pixels on the left screen L and parallax position X on the right screen R , calculated as well as (5), and the distance Z of the pixel:

[0009] (3);

[0010] Select the pixel with the smallest distance Z as the target pixel T, and call the current position of the target pixel T the original position. Set the target pixel displayed at the original position as T old , save the target pixel T old The original position data includes the target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old ;

[0011] Step 2: In response to the user's zoom-in instruction, the system obtains the target pixel T according to the user's zoom-in instruction old Zoom from the original position to the new position as the target pixel T new Vertical distance Z from the glasses new , and calculate the target pixel T new Parallax position X at the new position new-left and X new-right ;

[0012] Step 3: Calculate the parallax position X new-left and X new-right Then, first enlarge the entire screen of the left and right screens (X new / X old ) times, and then each pixel of the left screen enlarged image follows the target pixel T old From the original position X old-left Translate to new position X new-left , and each pixel of the right screen magnified image follows the target pixel T old From the original position X old-right Translate to new position X new-right .

[0013] The system also includes step 4: in response to the user's push-out instruction, the system obtains the new position X of the target pixel from step 3 according to the user's push-out instruction. new1 Arrived at the position after pushing away X new2 The vertical distance from the glasses, calculate the parallax position X of the target pixel reaching the position after the push according to the method in step 2 new2 , during the zooming process, first reduce the image on the left and right screens that have been enlarged in step 3 (X new2 / X new1 ) times, and then move each pixel of the reduced image in the left and right screens to follow the target pixel T from position X new1 Translate to new position X new2 .

[0014] The step 2 is specifically as follows:

[0015] In the left screen, the target pixel T is known old The vertical distance Z from the glasses at the original position old , target pixel T old Parallax position X displayed on the left screen old-left and target pixel T old Parallax position X displayed on the right screen old-left , using the simplified formula: X new =X old (Z old / Z new ) (4), we can calculate:

[0016] X new-left =X old-left (Z old / Z new ), X new-right =X old-right (Z old / Z new ).

[0017] The step 2 is specifically as follows:

[0018] Assuming that the target pixel T is located in the space area between the left and right center lines, the left screen will display the target pixel T to the right of the left center line, and the right screen will display the target pixel T to the left of the right center line. In response to the user's zoom-in instruction, the target pixel T at the original position is old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position oldThe display relationship of the target pixel T at the new position after zooming in new Parallax position X displayed on the right screen new-right and parallax angle , the parallax position X displayed on the left screen new-left and parallax angle , and the target pixel T new The vertical distance Z from the glasses at the new position new The display relationship is as follows:

[0019]

[0020] (6);

[0021] Since in the process of zooming in on the target pixel T, D L and D R If it remains unchanged,

[0022]

[0023] (7);

[0024] get: and ;

[0025] If the target pixel T old or T new If it falls on the left side of the left center line or the right side of the right center line, the left and right points L and R are taken as the origin respectively, and X takes a positive or negative number accordingly.

[0026] The target pixel T found in step 1 old The vertical distance Z from the glasses at the original position old If the preset value is exceeded, the system will old , target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle Get the given value.

[0027] The interaction instruction is issued through an auxiliary tool, which includes a control handle, a control bracelet or a control ring, or is given through a user's gesture.

[0028] A head-mounted display device, comprising a binocular camera for capturing a target image of a target area; the head-mounted display device also comprises a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement any of the above-mentioned methods for processing environmental image data in a camera perspective VST.

[0029] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for processing environmental image data in a camera perspective VST.

[0030] After adopting the technical solution of the present invention, when the system receives a zoom-in instruction from the user, it first searches for the pixel closest to the glasses as the target pixel, and obtains the parallax position X of the target pixel at the original position. old ; The system obtains the vertical distance between the new position of the target pixel after zooming in and the glasses according to the zoom-in instruction, and calculates the parallax position X of the target pixel at the new position new ; First, enlarge the entire screen on the left and right (X new / X old ) times, and then enlarge each pixel of the left and right screens to follow the target pixel from the parallax position X old Pan to parallax position X new . When the zoom-in function is activated, the target pixel moves toward the glasses, the screen image is enlarged and the vertical distance between the target pixel and the glasses is gradually reduced. When the zoom-out function is activated, the target pixel moves away from the glasses, the screen image is reduced and the vertical distance between the target pixel and the glasses is gradually increased. The present invention performs post-processing on the environmental image data in the camera perspective VST by software after the camera is mechanically focused, so that the user can immersively see the surrounding environment being zoomed in or zoomed out through the VST, can zoom in to view distant or too small objects, and achieve the visual effect of objects being zoomed in or zoomed out. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the display parallax relationship between the target pixel T in the surrounding environment and the left and right cameras with a focal length of f on the left and right screens seen through the VST method;

[0032] Figure 2 is the relationship curve between the target pixel T displayed at a distance of X mm from the center of the screen and the vertical distance Z between the target pixel T and the glasses when the left and right sides of formula (2) are used to calculate LOG at the same time;

[0033] Figure 3 A schematic diagram showing the parallax relationship during the process of zooming in and out of a target pixel of the present invention;

[0034] Figure 4A schematic diagram of finding the pixel closest to the glasses as the target pixel T in the present invention;

[0035] Figure 5 is the target pixel T at the original position in the first embodiment of the present invention. old Parallax position, parallax angle and vertical distance Z old Display relationship diagram;

[0036] Figure 6 is the target pixel T at the new position after zooming in the first embodiment of the present invention. new Parallax position, parallax angle and vertical distance Z new Display relationship diagram;

[0037] Figure 7 The functional structure block diagram of a head display device in the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0041] The principle of the present invention is described as follows:

[0042] like Figure 1As shown in FIG. 1 , this is a schematic diagram of the display parallax relationship of a target pixel T in the surrounding environment, and the left and right cameras L and R with a camera focal length of f on the left and right screens, where f is the optical focusing distance of the camera, that is, the imaging surface felt by the user. Assuming that the left and right screens of the glasses align the screen pixels according to the field of view (FOV) of the left and right cameras L and R, that is, the total number of pixels of the camera image is the same as the total number of pixels of the screen or the same after correction (for example, if the camera is 2000 and the screen is 2160, the glasses can choose to make the screen frame black with only 2000 pixels having VST perspective, or can choose to enlarge the camera image (2160 / 2000=1.08 times) for display. The correction here also refers to the ratio of pixel enlargement and reduction after the screen image of the present invention is enlarged or reduced), it is known that the total number of pixels on the x-axis of the screen is x. total and the field of view FOV of the left and right cameras; point L and point R are used to represent the center points of the left and right screens and the center points of the left and right cameras, and the line connecting point L and point R or the line parallel to them is set as the X axis. An XY coordinate system with point L and point R as the origin is established on the left and right screens, and the target pixel T is displayed on the imaging surface of the left and right screens at the left and right positions with parallax, respectively. The distance value between the parallax position of the target pixel T displayed on the left screen and point L is X L The right screen shows the distance between the disparity position of the target pixel T and point R, which is X R , the distance value X L and X R It is expressed in pixel values ​​of the left and right screens. In the real world, D is used to represent the distance between the target pixel T and point L or point R on the X-axis. The distance between the target pixel T and the normal of the glasses cutting the X-axis is D. L and right distance D R , where D L +D R =D IPD , the D IPD is the pupil distance of the left and right cameras, that is, the distance between point L and point R, which is a known fixed value. When the target pixel T is zoomed in and out, D L and D R The parallax angle is fixed and unchanged; when the objects on the left and right screens move from far to near, the parallax position is adjusted symmetrically from point L and point R along the X-axis toward the middle of the glasses. The normal line of point L is called the left center line, and the normal line of point R is called the right center line. and They respectively represent the degrees of the parallax position reached when the left / right center line is deviated toward the center of the glasses; since the processing of zooming in and out of the target pixel T in the left and right screens is the same, in order to facilitate the explanation of the principle, if the letter subscript does not distinguish between L and R, it means that it is applicable to both left and right screens.

[0043] like Figure 1As shown, the following formula is derived through triangle geometry calculation:

[0044] ZX L =D L f

[0045] ZX R =D R f(1);

[0046] The left and right cameras can be calculated separately and independently, and both can be converted using the following formula:

[0047] Z(X)=Df / X(2);

[0048] Where Z represents the vertical distance between the target pixel T and the glasses, X L and X R is the distance between the display position of the target pixel T on the left and right screens and the center of the screen, f is the camera focal length, D L and D R Represents the actual distance between the target pixel T and the center of the left and right screens on the X-axis of the glasses;

[0049] You can also use the parallax angle Calculate: and ;

[0050] Due to D L +D R =D IPD ,but (3);

[0051] like Figure 3 As shown in formula (2), when the target pixel T is moved closer or farther away, the distance Z changes when D L and D R The focusing distance f is a value that can only be changed when the lens is mechanically focused. The present invention is a software processing method after the mechanical focusing is completed, so the focusing distance f is also unchanged. Since the value Df is fixed, the simplified formula is: new X new =Df=Z old X old

[0052] X new =X old (Z old / Z new )(4).

[0053] like Figure 3 As shown, assuming T old is the target pixel at the original position, X old Display the target pixel T on the screenold The parallax position, Z old is the target pixel T old The vertical distance from the glasses at the original position; the target pixel T old After zooming in, it becomes the target pixel T at the new position new , the system infers Z based on user instructions new (The same value can be provided to the left and right screens for calculation). new is the target pixel T new The vertical distance from the glasses at the new position. old and Z new , X old and X new , D L and D R With the target pixel T old and T new From the triangular relationship, we can see that the closer the target pixel is to the glasses (T old >T new ), the smaller the vertical distance to the glasses (Z old >Z new ), the larger the value of X (X old <X new ); On the contrary, if the target pixel T is farther away from the glasses, the vertical distance Z from the glasses is larger, and the value of X is smaller.

[0054] The present invention adjusts and executes the user's zoom-in instruction according to the following steps:

[0055] Step 1: When the system receives the zoom-in instruction from the user, it searches for the pixel closest to the glasses as the target pixel T and obtains the parallax position X of the target pixel at the original position. old ;

[0056] Step 2: The system obtains the vertical distance between the new position of the target pixel after zooming in and the glasses according to the zoom-in instruction, and calculates the parallax position X of the target pixel at the new position new ;

[0057] Step 3: First, enlarge the entire screen of the left and right screens (X new / X old ) times, and then enlarge each pixel of the left and right screens to follow the target pixel from the parallax position X old Pan to parallax position X new .

[0058] Problem 1: Since the target pixel T is at the original position Z old and X old is unknown, so it is not possible to directly use formula (4) to calculate the new position X when receiving the pull-in command. new =X old(Z old / Z new );

[0059] Question 2: Assume that the pupil distance between the left and right cameras is D IPD is 100 mm, then the distance between the target pixel T and the center of the camera on the X-axis is D=50 mm, and the focal length of the camera is 50 mm. By calculating the LOG of both sides of formula (2), we can obtain Figure 2 The curve of the relationship between the target pixel T and the distance Z is shown at a distance of X mm from the center of the screen. In the process of enlarging or reducing the target pixel T, Df is a constant number, that is, Z multiplied by X is a fixed area. The relationship between them is a straight line when displayed with LOG. When the scene or image captured by the camera is infinitely far away, then X is 0, that is, if the target pixel T is at an infinite distance, then the target pixel T is displayed at point L and point R on the screen, that is, X L =X R =0 or ϴ L =ϴ R = 0. This means that in the process of searching for the pixel closest to the glasses as the target pixel T, if no target pixel T is found at a short distance or the distances Z of all pixels are infinite or close to infinity, then X old = 0, then it is impossible to use formula (4) to calculate the value X of the target pixel T displayed at the new position new .

[0060] Solution 1: Use the display screen pixel / resolution and camera field of view FOV provided by the manufacturer to calculate the Z of the target pixel T at the original position old and X old .

[0061] This paper uses open source models and tools to align the images seen by the left and right screens, so that each pixel can be compared. Each pixel has a corresponding parallax on the left and right screens. L and right X R Pixel position. Since the left and right screens of the glasses are aligned according to the field of view FOV of the left and right cameras, that is, the total number of pixels of the camera image is the same as the total number of pixels of the screen or the same after correction, the total number of pixels of the screen X axis is known to be X total , the field of view FOV and pupil distance D of the left and right cameras IPD , and PPD = X total / FOV, therefore, and (5) Use formula (3) to calculate the distance Z between the pixel and the glasses. Select the pixel with the smallest distance Z as the target pixel T, and obtain the vertical distance Z between the target pixel T and the glasses at the original position. old .

[0062] The open source code has many auxiliary depth calculation programs that can perform depth conversion on the images captured by the left and right cameras to obtain the depth color or grayscale value of each pixel in the left and right images, or obtain the depth color, grayscale value or depth data of each pixel in the left and right images based on the laser or TOF sensor. The existing technology for obtaining depth maps through binocular cameras is as follows: determine the internal parameters of the camera (such as focal length, principal point position, etc.) and external parameters (i.e., position and direction relative to the world coordinate system); collect image data of the calibration plate taken by the camera, and then use matlab or opencv binocular calibration of the camera pair, evaluate whether the reprojection error meets the requirements, detect the correctness of corner point extraction, and save all calibration results; use the calibration results to perform binocular correction on the image pairs taken by the same group of cameras, including image correction and cropping of the Region of Interest (ROI) area to obtain row-aligned left and right views; and calibrate the corrected left and right views. Figure 1 Generally, BM, SGBM, GC and other algorithms in OpenCV are used to estimate the depth and obtain the disparity map; the depth map is calculated based on the refined disparity map.

[0063] The present invention uses the existing image comparison technology to scan the left and right views collected by the camera line by line, compares all the pixels in each line, finds the pixels with the same depth information or grayscale color and the same arrangement in the left and right views, and obtains the parallax position X displayed by these pixels on the left screen. L and the parallax position X displayed on the right screen R , calculated as well as (5), and the vertical distance Z between the pixel and the glasses:

[0064] (3) ;

[0065] like Figure 4 As shown in FIG. 1 , the pixel closest to the glasses is selected as the target pixel T, the current position of the target pixel T is called the original position, and the target pixel displayed at the original position is T old , save the target pixel T old The original position data includes the target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old .

[0066] Find the target pixel Told After that, during the zooming process, the new parallax position X is calculated. new-left and X new-right There are two ways.

[0067] First, use the simplified formula (4): X new =X old (Z old / Z new ):

[0068] In response to the user's zoom-in instruction, in the left screen, the target pixel T old The vertical distance Z from the glasses at the original position old , target pixel T old Parallax position X displayed on the left screen old-left and target pixel T old Parallax position X displayed on the right screen old-right , according to the user's zoom instruction, Z new , calculate X new-left =X old-left (Z old / Z new ), X new-right =X old-right (Z old / Z new );

[0069] Second, assuming that the target pixel T is located in the space between the left and right center lines, the left screen will display the target pixel T to the right of the left center line, and the right screen will display the target pixel T to the left of the right center line. In response to the user's zoom-in instruction, according to Figure 5 Figure 6 As shown, the target pixel T at the original position old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old The display relationship of the target pixel T at the new position after zooming in new Parallax position X displayed on the right screen new-right and parallax angle , the parallax position X displayed on the left screen new-left and parallax angle , and the target pixel T new The vertical distance Z from the glasses at the new position new The display relationship is as follows:

[0070]

[0071] (6);

[0072] Since the target pixel T is in the process of zooming in, D L and D R If it remains unchanged,

[0073]

[0074] (7);

[0075] get: and ;

[0076] If the target pixel T old or T new If it falls on the left side of the left center line or the right side of the right center line, the left and right points L and R are taken as the origin respectively, and X takes a positive or negative number accordingly.

[0077] Under the premise that the position of the target pixel T relative to the user's real world remains unchanged, the present invention allows the user to feel that the target pixel T is being pulled closer or pushed away through software operation. Traditional mobile phones only have a monocular camera, and the object will not be felt to be pulled closer or pushed away when zooming in and out. Since smart glasses have binocular cameras and left and right display screens, the surrounding environment can be seen immersively through the VST method. If the visual effect of objects being pulled closer or pushed away is to be achieved, the parallax position of the target pixel T displayed on the left and right screens and the ratio of the entire image on the left and right screens must be changed. Therefore, the pixels of the entire image must be moved parallel to the target pixel T on the screen from the original position to the new position.

[0078] After calculating the parallax position X new-left and X new-right Then, first enlarge the entire screen of the left and right screens (X new / X old ) times, and then all pixels of the left screen magnified image follow the target pixel T from X old-left Pan to X new-left , and all pixels of the right screen magnified image follow the target pixel T from X old-right Pan to X new-right location.

[0079] Solution 2: In actual situations, if the vertical distance between the target pixel T and the glasses is too far, the target pixel T is zoomed in or out. X will be very small, and the user will not feel that the image is getting closer. At this time, it can be assumed that the vertical distance Z between the target pixel T and the glasses at the original position is oldis a relatively close preset value N, such as 5 meters or 5000mm, and assumes that the pupil distance D of the left and right cameras is IPD If the X axis of the target pixel T on the glasses is 100 mm and the distance between the center of the camera is D=50 mm, and the focal length of the camera is 50 mm, then X old =2500 / 5000=0.5. With these two assumptions, we can use formula (4) to scale up and adjust the new position X. new When the image is zoomed in, assuming it is from 5 meters away, the target pixel moves from Z old =5000mm Z pulled to the new position new When the image is enlarged (X new / X old ) times, and then the pixels of the entire image follow the center point of the image from X old Move parallel to new position X new That's it.

[0080] Embodiment 1

[0081] Embodiment 1 of the present invention provides a method for processing environmental image data in a camera perspective VST, which is applied to a system of an XR extended reality head-mounted display device. A binocular camera collects a real environment image, which is displayed on a screen of the head-mounted display device for viewing by a user. Based on an interactive instruction of the user, the system performs post-processing on the real environment image so that objects in the real environment image have a visual effect of zooming in and out. The interactive instruction includes a preset zoom-in instruction and a zoom-out instruction. The system predicts a zoom-in or zoom-out distance according to the interactive instruction. The real environment image refers to an environmental image of the real world collected by a binocular camera on a virtual reality device. The post-processing includes the following steps:

[0082] Step 1: In response to a zoom-in instruction given by the user when viewing the real environment image through the VST method, start looking for the pixel closest to the glasses as the target pixel T:

[0083] Assume that the left and right screens of the glasses are aligned according to the field of view FOV of the left and right cameras, that is, the total number of pixels of the camera image is the same as the total number of pixels of the screen or the same after correction. total , the field of view FOV of the left and right cameras, the pupil distance D of the left and right cameras IPD And the number of pixels per degree PPD = X total / FOV; Point L and point R are used to represent the center points of the left and right screens and the center points of the left and right cameras. The line connecting point L and point R or the line parallel to them is set as the X axis. An XY coordinate system with point L and point R as the origin is established on the left and right screens respectively. The distance between the parallax position of the target pixel T displayed on the left screen and point L is X LThe right screen shows the distance between the disparity position of the target pixel T and point R, which is X R , the distance value X L and X R Expressed in pixel values ​​on the left and right screens, when objects on the left and right screens move from far to near, the parallax positions are adjusted symmetrically from point L and point R along the X-axis toward the middle of the glasses. The normal line of point L is called the left center line, and the normal line of point R is called the right center line. The parallax angle is and They represent the degrees of parallax when the left / right center line is deviated toward the center of the glasses. In the real world, D represents the distance between the glasses and point L or point R on the X-axis. The distance between the target pixel T and the normal of the glasses cutting the X-axis is D. L and right distance D R , where D L +D R =D IPD , the D IPD is the pupil distance of the left and right cameras, that is, the distance between point L and point R, which is a known fixed value. When the target pixel T is zoomed in and out, D L and D R Fixed and unchanging;

[0084] Scan the left and right views captured by the camera line by line, compare all the pixels in each line, find the pixels with the same depth information or grayscale color and the same arrangement in the left and right views, and obtain the parallax position X of these pixels on the left screen L and the parallax position X on the right screen R , calculated as well as , and the distance Z of the pixel:

[0085] ;

[0086] Select the pixel with the smallest distance Z as the target pixel T, and call the current position of the target pixel T the original position. Set the target pixel displayed at the original position as T old , save the target pixel T old The original position data includes the target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old .

[0087] Step 2: In response to the user's zoom-in instruction, the system obtains the target pixel T according to the user's zoom-in instruction old Zoom from the original position to the new position as the target pixel T new Vertical distance Z from the glasses new , and calculate the target pixel T new Parallax position X at the new position new-left and X new-right :

[0088] Method 1: In the left screen, the target pixel T is known old The vertical distance Z from the glasses at the original position old , target pixel T old Parallax position X displayed on the left screen old-left and target pixel T old Parallax position X displayed on the right screen old-left , using the simplified formula: X new =X old (Z old / Z new ), we can calculate:

[0089] X new-left =X old-left (Z old / Z new ), X new-right =X old-right (Z old / Z new );

[0090] Method 2: Assuming that the target pixel T is located in the space between the left and right center lines, the left screen will display the target pixel T to the right of the left center line, and the right screen will display the target pixel T to the left of the right center line. In response to the user's zoom-in instruction, the target pixel T at the original position is displayed on the left screen. old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old The display relationship of the target pixel T at the new position after zooming in new Parallax position X displayed on the right screen new-right and parallax angle , the parallax position X displayed on the left screen new-left and parallax angle , and the target pixel T new The vertical distance Z from the glasses at the new positionnew The display relationship is as follows:

[0091]

[0092] (6);

[0093] Since in the process of zooming in on the target pixel T, D L and D R If it remains unchanged,

[0094]

[0095] (7);

[0096] get: and ;

[0097] If the target pixel T old or T new If it falls on the left side of the left center line or the right side of the right center line, the left and right origins L and R are taken as the origins, and X takes a positive or negative number accordingly;

[0098] It is possible that the calculated X value here has exceeded the screen size. Each glasses specification has FOV data. For example, if the screen has 100 FOV and 2000 pixels on the X axis, then the left and right 50 degrees have 1000 pixels or X. total / 2 value, then as long as the calculated X value exceeds ±1000 pixels, it cannot be displayed on the screen.

[0099] Step 3: Calculate X new-left and X new-right Then, first enlarge the entire screen of the left and right screens (X new / X old ) times, and then all pixels of the left screen magnified image follow the target pixel T from X old-left Pan to X new-left , and all pixels of the right screen magnified image follow the target pixel T from X old-right Pan to X new-right location.

[0100] like Figure 5 and 6 As shown, when the target pixel T old When zooming in to a new position, X old-left and X old-right Adjust to X new-left and X new-right , the pixel position of the real scene image should also be adjusted in proportion as a whole, so that X total-old =Xtotal , then X total-new =X total-old *(X old-left / X new-left ) = X total *( X old-left / X new-left ), PPD new =PPD old *(X old-left / X new-left ); Since the total number of pixels on the screen is still X total , so every (X new / X old ) pixel can only display one pixel of the real environment image. Assuming that the total number of pixels of the screen and the real environment image is the same = 2000, the field of view FOV is 100 degrees, PPD old is 20, if the total number of pixels of the enlarged image is X total-new =1000, then PPD new is 10, the total number of pixels on the screen is X total It is still 2000, which means that every 2 screen pixels can only display 1 pixel of the real environment image, which is twice the magnification.

[0101] Step 4: In response to the user's zoom instruction, the system obtains the new position X of the target pixel from step 3 according to the user's zoom instruction. new1 Arrived at the position after pushing away X new2 The vertical distance from the glasses, calculate the parallax position X of the target pixel reaching the position after the push according to the method in step 2 new2 , during the zooming process, first reduce the image on the left and right screens that have been enlarged in step 3 (X new2 / X new1 ) times, and then move each pixel of the reduced image in the left and right screens to follow the target pixel T from position X new1 Translate to new position X new2 .

[0102] In order to prevent the target pixel in step 4 from having black edges on the outer edge of the screen due to the image being reduced too small during the process of zooming out, the zooming out function can be stopped after the enlarged image is reduced to the size of the image captured by the original camera. Of course, the system can also allow the target pixel to continue to be zoomed out, then the image captured by the camera will be smaller than the screen, and at this time the visible image will appear smaller than the screen, that is, the image will be reduced and the surrounding area will be black or a new virtual image of the environment constructed by AIGC.

[0103] The target pixel T found in step 1 old The vertical distance Z from the glasses at the original position oldIf the preset value is exceeded, the system will old , target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle Get the given value.

[0104] The above-mentioned zoom in or zoom out instructions can be issued through auxiliary tools such as a control handle, a control bracelet or a control ring, or can be given through a user's gesture. The above-mentioned customized instructions belong to the prior art and are not the technical content of the present invention, and will not be repeated here.

[0105] The system predicts the distance to zoom in or out based on the interaction instruction. There are many existing technologies to achieve this. For example, it can be obtained by converting the duration given by the interaction instruction or by the amplitude of the user's gesture. This is not the technical content of the present invention and will not be elaborated here.

[0106] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] Specifically, each step of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor, or a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0108] Embodiment 2

[0109] A second embodiment of the present invention provides a head mounted display device, such as Figure 7As shown, the head display device 700 may include: a memory 710 and a processor 720, wherein the memory 710 is used to store a computer program and transmit the program code to the processor 720. In other words, the processor 720 may call and run the computer program from the memory 710 to implement the method in the first embodiment of the present application. For example, the processor 720 may be used to execute the method described in the first embodiment according to the instructions in the computer program.

[0110] In some embodiments of the present application, the processor 720 may include but is not limited to:

[0111] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0112] In some embodiments of the present application, the memory 710 includes, but is not limited to, volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAMbus RAM (DRRAM).

[0113] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to complete the method of the first embodiment provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the head mounted display device 700.

[0114] like Figure 7 As shown, the head display device may further include: a transceiver 730, which may be connected to the processor 720 or the memory 710. The processor 720 may control the transceiver 730 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices. The transceiver 730 may be at least two cameras for capturing target images of a target area.

[0115] It should be understood that the various components in the head display device 700 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0116] Embodiment 3

[0117] Embodiment 3 of the present invention further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer can execute the method in the above embodiment 1.

[0118] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing environmental image data in a camera perspective VST is applied to a system of an XR extended reality head-mounted display device, where a binocular camera collects real environmental images and displays them on the screen of the head-mounted display device for users to watch. The method is characterized by: Based on the user's interactive instructions, the system performs post-processing on the real environment image to make the objects in the real environment image have a visual effect of being zoomed in or pushed away. The interactive instructions include preset zoom-in instructions and push-away instructions. The system predicts the zoom-in or push-away distance according to the interactive instructions. The post-processing includes the following steps: Step 1: In response to a zoom-in instruction given by the user when viewing the real environment image through the VST method, start looking for the pixel closest to the glasses as the target pixel T: Assume that the left and right screens of the glasses are aligned according to the field of view FOV of the left and right cameras, that is, the total number of pixels of the camera image is the same as the total number of pixels of the screen or the same after correction. total , the field of view FOV of the left and right cameras, the pupil distance D of the left and right cameras IPD and pixels per degree ; Point L and point R are used to represent the center points of the left and right screens and the center points of the left and right cameras. The line connecting point L and point R or the line parallel to them is set as the X axis. An XY coordinate system with point L and point R as the origin is established on the left and right screens respectively. The distance between the parallax position of the target pixel T displayed on the left screen and point L is X L The right screen shows the distance between the disparity position of the target pixel T and point R, which is X R , the distance value X L and X R It is expressed in pixel values ​​of the left and right screens. When the objects on the left and right screens move from far to near, the parallax positions are adjusted symmetrically from point L and point R along the X-axis toward the middle of the glasses. The normal of point L is called the left center line, and the normal of point R is called the right center line. The parallax angle and They represent the degrees of the parallax position when the left / right center line is deviated toward the center of the glasses respectively; in the real world, D represents the actual distance between the target pixel T and point L or point R on the X-axis, and the distance between the target pixel T and the normal of the glasses cutting the X-axis is D L and right distance D R , where D L +D R =D IPD , the D IPD is the pupil distance of the left and right cameras, that is, the distance between point L and point R, which is a known fixed value. When the target pixel T is zoomed in and out, D L and D R Fixed and unchanging; Scan the left and right views captured by the camera line by line, compare all the pixels in each line, find the pixels with the same depth information or grayscale color and the same arrangement in the left and right views, and obtain the parallax position X of these pixels on the left screen L and the parallax position X on the right screen R , calculated as well as (5), and the distance Z of the pixel: （3）； Select the pixel with the smallest distance Z as the target pixel T, and call the current position of the target pixel T the original position. Set the target pixel displayed at the original position as T old , save the target pixel T old The original position data includes the target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old ; Step 2: In response to the user's zoom-in instruction, the system obtains the target pixel T according to the user's zoom-in instruction old Zoom from the original position to the new position as the target pixel T new Vertical distance Z from the glasses new , and calculate the target pixel T new Parallax position X at the new position new-left and X new-right ; Step 3: Calculate the parallax position X new-left and X new-right After that, first enlarge the entire screen of the left and right screens (X new / X old ) times, and then each pixel of the left screen enlarged image follows the target pixel T old From the original position X old-left Translate to new position X new-left , and each pixel of the right screen magnified image follows the target pixel T old From the original position X old-right Translate to new position X new-right .

2. The method for processing environmental image data in a camera perspective VST according to claim 1, characterized in that: The system also includes step 4: in response to the user's push-out instruction, the system obtains the new position X of the target pixel from step 3 according to the user's push-out instruction. new1 Arrived at the position X after pushing away new2 The vertical distance from the glasses, calculate the parallax position X of the target pixel reaching the position after the push according to the method in step 2 new2 , during the zooming process, first reduce the image on the left and right screens that have been enlarged in step 3 (X new2 / X new1 ) times, and then move each pixel of the reduced image in the left and right screens to follow the target pixel T from position X new1 Translate to new position X new2 .

3. The method for processing environmental image data in a camera perspective VST according to claim 1, characterized in that: The step 2 is specifically as follows: In the left screen, the target pixel T is known old The vertical distance Z from the glasses at the original position old , target pixel T old Parallax position X displayed on the left screen old-left and target pixel T old Parallax position X displayed on the right screen old-left , using the simplified formula: X new =X old (Z old / Z new ) (4), we can calculate: X new-left = X old-left (Z old / Z new ),X new-right = X old-right (Z old / Z new )。 4. The method for processing environmental image data in a camera perspective VST according to claim 1, characterized in that: The step 2 is specifically as follows: Assuming that the target pixel T is located in the space area between the left and right center lines, the left screen will display the target pixel T to the right of the left center line, and the right screen will display the target pixel T to the left of the right center line. In response to the user's zoom-in instruction, the target pixel T at the original position is old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle , and the target pixel T old The vertical distance Z from the glasses at the original position old The display relationship of the target pixel T at the new position after zooming in new Parallax position X displayed on the right screen new-right and parallax angle , the parallax position X displayed on the left screen new-left and parallax angle , and the target pixel T new The vertical distance Z from the glasses at the new position new The display relationship is as follows: （6）； Since in the process of zooming in on the target pixel T, D L and D R If it remains unchanged, （7）； get: and ; If the target pixel T old or T new If it falls on the left side of the left center line or the right side of the right center line, the left and right points L and R are taken as the origin respectively, and X takes a positive or negative number accordingly.

5. The method for processing environmental image data in a camera perspective VST according to claim 1, characterized in that: The target pixel T found in step 1 old The vertical distance Z from the glasses at the original position old If the preset value is exceeded, the system will old , target pixel T old Parallax position X displayed on the left screen old-left and parallax angle , target pixel T old Parallax position X displayed on the right screen old-right and parallax angle Get the given value.

6. The method for processing environmental image data in a camera perspective VST according to claim 1, characterized in that: The interaction instruction is issued through an auxiliary tool, which includes a control handle, a control bracelet or a control ring, or is given through a user's gesture.

7. A head mounted display device, characterized in that: The head-mounted display device includes a binocular camera for capturing a target image of a target area; the head-mounted display device also includes a memory and a processor, the memory is used to store a computer program; the processor is used to execute the computer program to implement the method for processing environmental image data in a camera perspective VST in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the method for processing environmental image data in a camera perspective VST according to any one of claims 1 to 6.

Images