Virtual window configuration apparatus, method and system

Abstract

A virtual window configuration method includes generating a virtual window by a processor, calculating a depth configuration block or a feature point configuration block by the processor, and moving the virtual window to the depth configuration block or the feature point configuration block by the processor. Through depth detection and / or feature point detection, a target block that has the least impact on user operation can be automatically found, and the virtual window is moved to the target block, thereby reducing the problem of occlusion and the purpose of user manual operation.

Description

Technical Field

[0001] This invention relates to a window configuration device, and more particularly to a virtual window configuration device, a virtual window configuration method, and a virtual window configuration system. Background Technology

[0002] Augmented Reality (AR) technology can present virtual information through augmented reality glasses, combining it with real-world scenes to provide users with an "immersive experience." AR technology is often used in teaching, training, and remote assistance.

[0003] Due to the limited field-of-view (FOV) of augmented reality (AR) glasses, it's impossible to present all information to the user simultaneously, especially in remote expert assistance applications. AR glasses must display a multitude of information at once, such as call windows, document windows, and message windows. Users often need to drag virtual windows outside the main workspace individually and turn their heads to view the virtual information, thus avoiding obstructing the main workspace.

[0004] Therefore, how to automatically move the virtual window to a suitable position without obscuring the main workspace when it is reopened has become one of the problems that need to be improved in this field. Summary of the Invention

[0005] To address the aforementioned problems, one embodiment of the present invention provides a virtual window configuration device. The virtual window configuration device includes: a processor, a depth detection sensor, a feature point detection sensor, and a storage device. The processor generates a virtual window. The depth detection sensor generates depth information based on an image. The feature point detection sensor generates feature point information based on the image. The processor accesses programs stored in the storage device to implement a depth analysis module and a feature point analysis module, wherein: the depth analysis module analyzes the depth information to generate a depth matrix and identifies a depth configuration block in the image based on the depth matrix; the feature point analysis module analyzes the feature point information to generate a feature point matrix and identifies a feature point configuration block in the image based on the feature point matrix. The processor moves the virtual window to the depth configuration block or moves the virtual window to the feature point configuration block.

[0006] To address the aforementioned problems, one embodiment of the present invention provides a virtual window configuration method, comprising: generating a virtual window using a processor; generating depth information based on an image using a depth detection sensor; wherein the processor analyzes the depth information to generate a depth matrix, and identifies a depth configuration block in the image based on the depth matrix; generating feature point information based on the image using a feature point detection sensor, analyzing the feature point information using the processor to generate a feature point matrix, and identifying a feature point configuration block in the image based on the feature point matrix; and moving the virtual window from a viewpoint to the depth configuration block, or moving the virtual window from a viewpoint to the feature point configuration block, using the processor.

[0007] To address the aforementioned problems, one embodiment of the present invention provides a virtual window configuration system comprising: a processor and augmented reality (AR) glasses. The processor moves a virtual window to a line-of-sight position in an image. The AR glasses include: a depth sensor and a feature point sensor. The depth sensor generates depth information based on an image. The feature point sensor generates feature point information based on the image. The AR glasses transmit the depth information and feature point information to the processor. The processor analyzes the depth information to generate a depth matrix and identifies a depth configuration block in the image based on the depth matrix. The processor analyzes the feature point information to generate a feature point matrix and identifies a feature point configuration block in the image based on the feature point matrix. The processor moves the virtual window to the depth configuration block or moves the virtual window to the feature point configuration block.

[0008] The virtual window configuration device, virtual window configuration method, and virtual window configuration system shown in the embodiments of the present invention provide a method for dynamically moving virtual windows on a virtual reality device. By using depth detection and / or feature point detection, a target block that will not affect the user's operation can be automatically found, and the virtual window can be moved to this target block, thereby reducing occlusion problems and the need for manual operation by the user. Attached Figure Description

[0009] Figure 1 This is a block diagram of a virtual window configuration device according to an embodiment of the present invention.

[0010] Figure 2 This is a flowchart illustrating a virtual window configuration method according to an embodiment of the present invention.

[0011] Figure 3 This is a schematic diagram illustrating a virtual window configuration method according to an embodiment of the present invention.

[0012] Figure 4 This is a schematic diagram illustrating depth information according to an embodiment of the present invention.

[0013] Figure 5 This is a schematic diagram of a depth matrix according to an embodiment of the present invention.

[0014] Figure 6 This is a schematic diagram illustrating feature point information according to an embodiment of the present invention.

[0015] Figure 7 This is a schematic diagram of a feature point matrix according to an embodiment of the present invention.

[0016] Figure 8 This is a schematic diagram of a movable virtual window according to an embodiment of the present invention.

[0017] Figure 9 This is a schematic diagram of a movable virtual window according to an embodiment of the present invention.

[0018] Figure 10 This is a flowchart illustrating a method for moving a virtual window based on depth information according to an embodiment of the present invention.

[0019] Figure 11 It is a flowchart of moving the virtual window based on feature point information.

[0020] Figure 12 This is a flowchart illustrating a method for moving a virtual window based on depth information and feature point information, according to an embodiment of the present invention.

[0021] Figure 13A and Figure 13B This is a flowchart illustrating a method for moving a virtual window based on depth information and feature point information, according to an embodiment of the present invention.

[0022] Among them, the attached figures are labeled

[0023] 100 Virtual Window Configuration Device

[0024] 200 Virtual Window Configuration Method

[0025] 300 Virtual Window Configuration Method

[0026] 30 Augmented Reality Glasses

[0027] 32 Main Workspace

[0028] USR users

[0029] 33 Instructors

[0030] ARW Virtual Window

[0031] 10 processors

[0032] 12 Depth Detection Sensors

[0033] 14 Feature Point Detection Sensor

[0034] 16 Storage devices

[0035] 18. Deep Analysis Module

[0036] 20 Feature Point Analysis Module

[0037] Steps 210~240, 101~106, 111~116, 121~128, 131~142 Detailed Implementation

[0038] The following description is a preferred embodiment of the invention and is intended to describe the basic spirit of the invention, but is not intended to limit the invention. The actual scope of the invention must be understood by referring to the claims below.

[0039] It must be understood that the words “comprising” and “including” used in this specification are used to indicate the presence of specific technical features, values, method steps, work processes, elements and / or components, but do not preclude the addition of more technical features, values, method steps, work processes, elements, components, or any combination thereof.

[0040] In patent applications, the use of terms such as "first," "second," and "third" to modify elements in the patent application is not intended to indicate a priority order, a prior relationship, or that one element precedes another, or the chronological order of the execution of method steps; it is only used to distinguish elements with the same name.

[0041] Please refer to the above as well. Figures 1 to 2 , Figure 1 This is a block diagram of a virtual window configuration device 100 according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a virtual window configuration method 200 according to an embodiment of the present invention.

[0042] In one embodiment, such as Figure 1 As shown, the virtual window configuration device 100 includes a processor 10, a depth sensor 12, a feature point sensor 14, and a storage device 16.

[0043] In one embodiment, the storage device 16 stores a depth analysis module 18 and a feature point analysis module 20. In another embodiment, the processor 10 accesses the program stored in the storage device 16 to implement the functions of the depth analysis module 18 and the feature point analysis module 20.

[0044] In one embodiment, the depth analysis module 18 and the feature point analysis module 20 may be programs stored in the storage device 16. In one embodiment, the depth analysis module 18 and the feature point analysis module 20 may be implemented by firmware. In one embodiment, the depth analysis module 18 and the feature point analysis module 20 may be implemented by circuitry, chips, or other hardware, and each may be electrically coupled to the storage device.

[0045] In one embodiment, the depth detection sensor 12, the feature point detection sensor 14, and the storage device 16 are each electrically coupled to the processor 10.

[0046] In one embodiment, the processor 10, depth sensor 12, feature point sensor 14, and storage device 16 are located on an augmented reality device.

[0047] In one embodiment, the virtual window configuration device 100 can be applied to an augmented reality device, such as augmented reality glasses 30, a mobile phone, or other devices capable of applying augmented reality. For ease of explanation, the following specification uses augmented reality glasses 30 as an example.

[0048] In one embodiment, the virtual window configuration device 100 further includes a gravity sensor (G-Sensor) for detecting a turning position of the augmented reality glasses 30.

[0049] In one embodiment, the virtual window configuration device 100 further includes a camera for capturing images of the actual scene by turning towards the desired position. In one embodiment, the processor 10 calculates the viewing position of the virtual window in the image (e.g., the position the user is viewing) based on the turning position. In one embodiment, the camera is implemented using a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS).

[0050] In one embodiment, the processor 10 may be implemented by an integrated circuit such as a microcontroller, microprocessor, digital signal processor (DSP), field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or a logic circuit.

[0051] In one embodiment, commonly used depth detection sensors 12 include two-dimensional lidar, three-dimensional lidar, stereoscopic cameras, time-of-flight (ToF) cameras, radio detection and ranging (RADAR), and ultrasonic radar, etc. Except for stereoscopic cameras which use binocular vision technology for ranging, the other sensors all use time-of-flight ranging technology. However, the present invention is not limited to this; any sensor capable of measuring the depth at various locations in an image can be used as the depth detection sensor 12.

[0052] In one embodiment, the feature point detection sensor 14 is, for example, a surface photoelectric sensor, an image recognition sensor, or a vision sensor. However, the present invention is not limited to these; any sensor capable of detecting feature points in an image can be used as the feature point detection sensor 14. In one embodiment, the feature point detection sensor 14 is integrated with a camera into a camera module, using the captured image to detect feature points in the image.

[0053] In one embodiment, the depth sensor 12, the feature point sensor 14, and another processor may be included in an augmented reality glasses 30. The processor 10 may be located on another electronic device (e.g., a computer, server, or other electronic device with computing and storage functions), and the augmented reality glasses 30 is electrically coupled to this other electronic device. In this example, the augmented reality glasses 30 and the other electronic device are considered as a virtual window configuration system.

[0054] In one embodiment, the storage device 16 may be implemented by a read-only memory, flash memory, floppy disk, hard disk, optical disk, USB flash drive, magnetic tape, a network-accessible database, or a storage medium with the same function.

[0055] Please see Figures 2 to 3 The virtual window configuration method 200 can be implemented by the virtual window configuration device 100 or the virtual window configuration system described above. Figure 3 This is a schematic diagram of a virtual window configuration method 300 according to an embodiment of the present invention.

[0056] In step 210, the processor 10 moves a generated virtual window to a line-of-sight position.

[0057] In one embodiment, for example Figure 3As shown, if a user (USR) needs remote assistance, they wear augmented reality glasses 30 to see the main work area 32, which is a real-world scene. In this example, the main work area 32 contains some wiring. The display of the augmented reality glasses 30 shows a virtual ARW (Archived View). The virtual ARW can be used to remotely guide the user (USR) on how to repair the wiring in the main work area 32. For example, the virtual ARW shows an instructor 33 demonstrating how to repair the wiring. When the virtual ARW is displayed, the processor 10 moves it according to a preset method to the image displayed on the augmented reality glasses 30, to the user (USR)'s line of sight.

[0058] However, this would cause the main workspace 32 to overlap with the virtual window ARW, resulting in the user's USR main workspace 32 being blocked by the virtual window ARW. Therefore, in subsequent steps, it is necessary to find an appropriate space in the image to place the virtual window ARW.

[0059] In step 220, the depth detection sensor 12 generates depth information based on an image; wherein, the processor 10 analyzes the depth information to generate a depth matrix, and finds a depth configuration block in the image based on the depth matrix.

[0060] Please see Figures 4 to 5 , Figure 4 This is a schematic diagram illustrating depth information according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a depth matrix according to an embodiment of the present invention.

[0061] like Figure 4 As shown, the depth detection sensor 12 (e.g., a time-of-flight ranging sensor) uses a light-emitting diode to emit infrared light, which is reflected back from the object to generate depth information. The framed area is a space that is relatively far away from the user and large enough to fit into the virtual view window (ARW).

[0062] Therefore, we can obtain the depth information of each pixel using a time-of-flight ranging sensor, store the depth information in an array, and scan from left to right and top to bottom to find the deeper blocks. The depth sensor 12 and / or processor 10 can generate [data / data] based on the depth information. Figure 5 The depth matrix in the matrix.

[0063] In one embodiment, Figure 5 Each cell in the depth matrix is ​​a pixel, and the numerical unit is, for example, meters.

[0064] In one embodiment, the threshold value for setting the block size requires a minimum of 5×3 pixels.

[0065] In one embodiment, the virtual viewport (ARW) can be scaled.

[0066] In one embodiment, the processor 10 may read a preset block in the storage device 16, for example, 5×3 pixels, and use 5×3 pixels as a threshold value for setting the block size.

[0067] In one embodiment, the preset block may be the same size as the virtual window ARW, or a block with the same aspect ratio as the virtual window ARW.

[0068] For example, assuming the virtual viewport ARW corresponds to the size of the depth information, and a block that meets the depth threshold needs to be at least 5×3 pixels, then to find a block with a horizontal axis greater than 5 pixels and a vertical axis greater than 3 pixels, the following steps are required.

[0069] Processor 10 first checks the deepest depth value in the depth matrix and compares whether the size of this block is greater than 5×3 pixels. In this case, the deepest depth value is 0.8 (in this case, the larger the depth value, the denser the diagonal lines), and the block size of the deepest depth value is 1×4, which does not meet the 5×3 pixel requirement.

[0070] Next, continue searching downwards based on the depth value to find the block with a depth of 0.7. This block can be further divided into candidate blocks of 14×4, 3×7, and / or 3×7. Candidate blocks must be rectangles or squares.

[0071] Among them, the block that meets the 5×3 pixel requirement is a candidate block of 14×4. The processor 10 then searches downwards according to the depth value and finds a block with a depth value of 0.6. The size of the candidate block is 3×7 pixels. It does not meet the 5×3 pixel requirement and is smaller than the 14×4 block with a depth value of 0.7. Therefore, the processor 10 stops searching.

[0072] In one embodiment, if only the depth value is used for judgment, a 14×4 block with a depth of 0.7 is set as the depth configuration block. Then, according to the aspect ratio (16:9) of the virtual window ARW, the aspect ratio of the virtual window ARW is calculated to be 1.77 times. Therefore, the actual size of the virtual window ARW that can be placed in the 14×4 block is (4*1.77)×4=7×4, which is a candidate block size. Finally, based on the coordinate position of the time-of-flight ranging sensor and the corresponding pixels, the three-dimensional spatial coordinates and size of this candidate block can be calculated, and the virtual window ARW is moved to this three-dimensional coordinate (the position of the depth configuration block).

[0073] In one embodiment, the depth value of each pixel in the depth configuration block is greater than a depth threshold (e.g., greater than 0.5; therefore, in the above embodiment, only the block size with a depth value of 0.6 to 0.8 is analyzed). In one embodiment, the area of ​​the depth configuration block is larger than the virtual viewport ARW.

[0074] In one embodiment, the processor 10 adjusts the virtual window ARW to fit the size of the depth configuration block based on an original aspect ratio of the virtual window ARW, and moves the virtual window ARW from the line-of-sight position to the depth configuration block.

[0075] In step 230, the feature point detection sensor 14 generates feature point information based on the image, and the processor 10 analyzes the feature point information to generate a feature point matrix, and finds a feature point configuration block in the image based on the feature point matrix.

[0076] Please see Figures 6 to 7 , Figure 6 This is a schematic diagram illustrating feature point information according to an embodiment of the present invention. Figure 7 This is a schematic diagram of a feature point matrix according to one embodiment of the present invention.

[0077] In one embodiment, the processor 10 uses a known image feature detection algorithm to determine which pixels in the entire image contain feature point information (such as...). Figure 6 As shown), this feature point information is stored in a feature point array for scanning (e.g. Figure 7 As shown), find the blocks without feature points (such as...). Figure 6 (The part outlined by the dashed line).

[0078] In one embodiment, known image feature detection algorithms can use graphs to find feature points such as corners, blocks, and sharp areas in an image. These feature points can be displayed in a dotted manner. Figure 6 Therefore, we can see the blocks without feature points and select these blocks.

[0079] Therefore, we can obtain the feature point information of each pixel using known image feature detection algorithms, store the feature point information in a feature point array, and scan from left to right and top to bottom to find blocks where the number of feature points is less than a feature point threshold value. The feature point threshold value can be, for example, 0, 1, or 2. The processor 10 determines whether the sum of feature points in the block is less than the feature point threshold value and finds the largest block that meets the condition of being less than the feature point threshold value. The feature point detection sensor 14 and / or processor 10 can generate [data / information] based on the feature point information. Figure 7 The feature point array in the image.

[0080] In one embodiment, Figure 7 Each cell in the feature point array is a pixel, and the value in each cell is the feature value. For example, 0 means there is no feature point, and 1 means there is a feature point.

[0081] For example, assuming the virtual viewport ARW corresponds to the size of the feature point information, the preset block needs to be at least 5×3 pixels. Therefore, to find a block with a horizontal axis greater than 5 pixels and a vertical axis greater than 3 pixels, the following steps are required.

[0082] Processor 10 first identifies blocks with no feature points (i.e., feature values ​​of 0 in the pixel grid), and then finds the largest block. Figure 7 In the example, it is an 11×4 block.

[0083] After confirming that the 11×4 block is larger than the required 5×3 pixels, the processor 10 sets the 11×4 block (indicated by diagonal lines at the top left and bottom right) as a candidate block. In one embodiment, the candidate block must be rectangular or square. In this example, the size of the candidate block is greater than or equal to the required preset block size, such as 5×3 pixels. Therefore, the processor 10 sets the candidate block as a feature point configuration block.

[0084] In response to the setting that only feature point information is used for judgment, the matching 11×4 block is set as a candidate block. Then, according to the aspect ratio of the virtual window (16:9), the aspect ratio of the virtual window ARW is calculated to be 1.77 times. Therefore, the actual size of the virtual window ARW that can be placed in the 11×4 block is 7×4. Finally, based on the camera's coordinate position and the corresponding pixels, the three-dimensional spatial coordinates and size of this candidate block can be calculated, and the virtual window ARW is moved to this three-dimensional coordinate (configuring the block for the feature point).

[0085] One method involves using the camera's coordinates and corresponding pixels. For example, the virtual window occupies 7×4 pixels within the field of view (FOV). Based on the known FOV calculation formula, when the processor 10 obtains the camera's focal length and the distance from the camera to the object (i.e., depth information), it can deduce the actual size of the virtual window (i.e., the field of view). The three-dimensional spatial coordinates are based on the current three-dimensional coordinates of augmented reality (AR) glasses. Adding the depth information, the three-dimensional coordinates of the object in front of the camera can be used to deduce the three-dimensional coordinates of the object in front of the camera.

[0086] In one embodiment, the number of feature points in the feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than a preset block.

[0087] In one embodiment, the processor 10 adjusts the virtual window ARW to fit the size of the feature point configuration block based on an original aspect ratio of the virtual window ARW, and moves the virtual window ARW from the line of sight position to the feature point configuration block.

[0088] In step 240, the processor 10 moves the virtual viewport ARW from the viewpoint to the depth configuration block, or moves the virtual viewport ARW from the viewpoint to the feature point configuration block.

[0089] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating a movable virtual viewport (ARW) according to an embodiment of the present invention. In one embodiment, when a depth configuration block has been calculated, the processor 10 can move the virtual viewport ARW from the viewpoint to the depth configuration block. In one embodiment, when a feature point configuration block has been calculated, the virtual viewport ARW is moved from the viewpoint to the feature point configuration block. In one embodiment, when both the depth configuration block and the feature point configuration block have been calculated, the processor 10 can take the overlapping block (i.e., the intersection block) between them. If this intersection block is larger than the pixels required by the virtual viewport ARW, the virtual viewport ARW is moved to this intersection block. In one embodiment, the processor transmits the coordinates of the overlapping block to the augmented reality glasses; another processor of the augmented reality glasses moves the virtual viewport to the overlapping block based on the coordinates of the overlapping block.

[0090] In one embodiment, the depth value of each pixel in the depth configuration block is greater than a depth threshold value, and the area of ​​the depth configuration block is greater than or equal to a preset block. The number of feature points in the feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block. The processor 10 simultaneously analyzes the depth configuration block and the feature point configuration block to find an overlapping block and determines whether the area of ​​the overlapping block is greater than or equal to the preset block. In response to the processor 10 determining that the area of ​​the overlapping block is greater than the preset block, the processor 10 moves the virtual view window ARW from the view position to the overlapping block.

[0091] In one embodiment, in response to the processor 10 determining that the area of ​​the overlapping block is not greater than or equal to a preset block, the processor 10 does not move the position of the virtual view window ARW, or moves the virtual view window ARW to the depth configuration block or the feature point configuration block.

[0092] In one embodiment, if the processor 10 cannot find a depth configuration block in which the depth value of each pixel is greater than a depth threshold and the area is greater than or equal to a preset block, then the processor 10 searches for a feature point configuration block in which the number of feature points is less than a feature point threshold and the area of ​​the feature point configuration block is greater than or equal to the preset block, and moves the virtual viewport ARW to the feature point configuration block. In another embodiment, if the processor 10 cannot find a depth configuration block in which the depth value of each pixel is greater than a depth threshold and the area is greater than or equal to the preset block, nor can it find a feature point configuration block in which the number of feature points is less than a feature point threshold and the area is greater than or equal to the preset block, then the virtual viewport ARW remains in its original position.

[0093] In one embodiment, in response to the processor 10 determining that the area of ​​the overlapping block is greater than or equal to a preset block, the processor 10 adjusts the virtual window ARW to fit the size of the overlapping block according to an original aspect ratio of the virtual window ARW, and moves the virtual window ARW from the line of sight to the overlapping block.

[0094] Please see Figure 9 , Figure 9 This is a schematic diagram of a mobile virtual window (ARW) according to an embodiment of the present invention.

[0095] If Figure 7 Selected feature points are configured to be overlaid into blocks. Figure 5 The deep configuration block can then be obtained Figure 9 The overlapping blocks are marked with diagonal lines from the top right to the bottom left and from the top left to the bottom right.

[0096] In response to the processor 10 determining the area of ​​the overlapping blocks (e.g. Figure 9 If the area of ​​the overlapping block in the virtual window ARW is greater than or equal to the preset block, the processor 10 adjusts the virtual window ARW to fit the size of the overlapping block according to the original aspect ratio of the virtual window ARW, and moves the virtual window ARW from the line of sight to the overlapping block.

[0097] In one embodiment, the overlapping area is the optimal placement position for the virtual viewport (ARW).

[0098] Please see Figures 10 to 13A and Figure 13B , Figure 10 This is a flowchart illustrating a motion virtual window (ARW) based on depth information, according to an embodiment of the present invention. Figure 11 This is a flowchart illustrating a virtual window ARW that moves based on feature point information, according to an embodiment of the present invention. Figure 12Figure 13 is a flowchart illustrating a virtual viewport (ARW) based on depth information and feature point information according to an embodiment of the present invention.

[0099] Figure 10 This is a flowchart illustrating a motion virtual window (ARW) based on depth information, according to an embodiment of the present invention.

[0100] In step 101, a turning position of an augmented reality glasses 30 is detected by a gravity sensor, and an image is captured by a camera facing the turning position. The image contains an actual scene. The processor 10 calculates the viewing position of the virtual window ARW in the image based on the turning position, and the processor 10 first follows the virtual window ARW to a viewing position.

[0101] In one embodiment, the line of sight may be the position in front of the user (USR).

[0102] In step 102, the depth detection sensor 12 acquires depth information in front of the augmented reality device.

[0103] In one embodiment, the augmented reality device may be augmented reality glasses 30, a mobile phone, or other devices capable of applying augmented reality. For ease of explanation, in the following... Figures 10 to 1 In the embodiment of 3, augmented reality glasses 30 are used as an example.

[0104] In step 103, the processor 10 analyzes the depth information through the depth detection sensor 12 to find a block that is deep and large in area.

[0105] In one embodiment, a deeper block refers to a block that is farther away from the augmented reality glasses 30 and the distance is greater than the depth threshold value.

[0106] In step 104, the processor 10 uses the depth detection sensor 12 to determine whether a block with a relatively deep depth and an area greater than or equal to a preset block has been found. If yes, proceed to step 105; otherwise, proceed to step 106.

[0107] In one embodiment, the depth detection sensor 12 can transmit depth information to the processor 10, which can then determine whether a block with a greater depth and an area greater than or equal to a preset block has been found.

[0108] In one embodiment, the processor 10 considers blocks with greater depth and an area greater than or equal to a preset block as candidate blocks.

[0109] In step 105, the processor 10 considers one of the candidate blocks as a deep configuration block (also known as the target block), adjusts the virtual viewport ARW to a suitable size according to the size of this deep configuration block, and moves the virtual viewport ARW to the deep configuration block. In one embodiment, the processor 10 may select the block with the largest area from multiple candidate blocks as the deep configuration block.

[0110] In one embodiment, assuming the virtual viewport ARW requires a size of at least 5×3 pixels, it is necessary to find a block with a horizontal axis greater than 5 pixels and a vertical axis greater than 3 pixels. The processor 10 sets a block that meets the depth (e.g., 0.7 meters) (e.g., a size of 14×4) as a depth configuration block. Then, based on the aspect ratio of the virtual viewport ARW (e.g., 16:9), it calculates that the aspect ratio of the virtual viewport ARW is 1.77 times. Therefore, within the 14×4 block, the actual size of the candidate block that can accommodate the virtual viewport ARW is (4*1.77)×4 = 7×4. Finally, based on the coordinate position of the time-of-flight ranging sensor and the corresponding pixels, the three-dimensional spatial coordinates and size of the candidate block can be calculated, and the virtual viewport ARW is moved to this three-dimensional coordinate (the position of the depth configuration block). This adjusts the virtual viewport ARW to a suitable size.

[0111] In step 106, the processor 10 keeps the virtual window ARW in place.

[0112] Next, please refer to Figure 11 , Figure 11 This is a flowchart of the ARW (Action View Window) based on feature point information.

[0113] In step 111, a turning position of an augmented reality glasses 30 is detected by a gravity sensor, and an image is captured by a camera facing the turning position; wherein, the image contains an actual scene, and the processor 10 calculates the viewing position of the virtual window ARW in the image based on the turning position, and the processor 10 first follows the virtual window ARW to a viewing position.

[0114] In step 112, the feature point detection sensor 14 acquires feature point information in front of the augmented reality device (such as augmented reality glasses 30).

[0115] In step 113, the processor 10 analyzes the feature point information through the feature point detection sensor 14 to find blocks with low feature points and large areas.

[0116] In one embodiment, a large area block refers to a block with an area greater than or equal to a preset block. A low feature point block refers to a block in which the number of feature points is less than a feature point threshold value.

[0117] In step 114, the processor 10 uses the feature point detection sensor 14 to determine whether a large area with low feature points has been found. If yes, proceed to step 115; otherwise, proceed to step 116.

[0118] In one embodiment, the feature point detection sensor 14 can transmit feature point information to the processor 10, and the processor 10 can determine whether a large area with low feature points has been found.

[0119] In step 115, the processor 10 regards the low-feature-point and large-area block as the feature-point configuration block (i.e., the target block), adjusts the virtual window ARW to a suitable size according to the size of the feature-point configuration block, and moves the virtual window ARW to the feature-point configuration block.

[0120] Therefore, the number of feature points in a feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block.

[0121] In one embodiment, assuming that the virtual window ARW corresponds to a feature point information size of at least 5×3 pixels, it is necessary to find a block with a horizontal axis greater than 5 pixels and a vertical axis greater than 3 pixels. In response to the setting of judging based on feature point information, the 11×4 block that meets the criteria is set as a candidate block. Then, according to the aspect ratio (16:9) of the virtual window, the aspect ratio of the virtual window ARW is calculated to be 1.77 times. Therefore, the actual size of the virtual window ARW that can be placed in the 11×4 block is 7×4. Finally, based on the coordinate position of the camera and the corresponding pixels, the three-dimensional spatial coordinates and size corresponding to this candidate block can be calculated, and the virtual window ARW is moved to this three-dimensional coordinate (configuring a block for feature points).

[0122] In step 116, the processor 10 keeps the virtual window ARW in place.

[0123] Next, please refer to Figure 12 , Figure 12 This is a flowchart of the ARW (Action View Window) based on depth information and feature point information.

[0124] In step 121, a turning position of an augmented reality glasses 30 is detected by a gravity sensor, and an image is captured by a camera facing the turning position. The image contains an actual scene. The processor 10 calculates the viewing position of the virtual window ARW in the image based on the turning position, and the processor 10 first follows the virtual window ARW to a viewing position.

[0125] In step 122, the depth detection sensor 12 acquires the depth information in front of the augmented reality glasses 30, and the feature point detection sensor 14 acquires the feature point information in front of the augmented reality glasses 30.

[0126] In step 123, the processor 10 analyzes the depth information through the depth detection sensor 12 to find a block that is deep and large in area.

[0127] In one embodiment, a deeper block refers to a block that is far from the augmented reality glasses 30 and the distance is greater than the depth threshold value, and such a block is called a depth configuration block.

[0128] In step 124, the processor 10 analyzes the feature point information through the feature point detection sensor 14 to find blocks with low feature points and large areas, and calls such blocks feature point configuration blocks.

[0129] In one embodiment, a large area block refers to a block with an area greater than or equal to a preset block. A low feature point block refers to a block in which the number of feature points is less than a feature point threshold value.

[0130] Steps 123 and 124 can be executed sequentially, out of order, or in parallel.

[0131] In step 125, the processor 10 simultaneously analyzes the depth configuration block and the feature point configuration block to find overlapping blocks.

[0132] In step 126, the processor 10 determines whether a block with a relatively deep depth and low feature points has been found. If yes, proceed to step 127; otherwise, proceed to step 128.

[0133] In step 127, the processor 10 considers the block with deep depth and low feature points as the target block, adjusts the virtual window ARW to a suitable size according to the size of the target block, and moves the virtual window ARW to the target block.

[0134] In step 128, the processor 10 keeps the virtual window ARW in place.

[0135] Next, please refer to Figures 13A to 13B , Figures 13A to 13B This is a flowchart of the ARW (Action View Window) based on depth information and feature point information. Figures 13A to 13B Steps 131 to 137 in the text are respectively related to Figure 12 Steps 121 to 127 are the same, so they will not be repeated here. Figures 13A to 13B If the determination in step 136 is negative, proceed to step 138.

[0136] In step 138, the processor 10 determines whether a deeper block has been found. If yes, proceed to step 139; otherwise, proceed to step 140.

[0137] In one embodiment, a deeper block refers to a depth configuration block, in which the depth value of each pixel is greater than a depth threshold value, and the area of ​​the depth configuration block is greater than or equal to a preset block.

[0138] In step 139, the processor 10 regards the deeper block as the target block, adjusts the virtual window ARW to a suitable size according to the size of the target block, and moves the virtual window ARW to the target block.

[0139] In step 140, the processor 10 determines whether a low-feature block has been found. If yes, proceed to step 141; otherwise, proceed to step 142.

[0140] In one embodiment, a low feature point block refers to a block in which the number of feature points is less than a feature point threshold value.

[0141] In one embodiment, a low-feature block is considered a feature point configuration block, wherein the number of a feature point in the feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to a preset block.

[0142] In step 141, the processor 10 regards the low feature point block as the target block, adjusts the virtual window ARW to a suitable size according to the size of the target block, and moves the virtual window ARW to the target block.

[0143] In step 142, the processor 10 keeps the virtual window ARW in place.

[0144] The virtual window configuration device, virtual window configuration method, and virtual window configuration system shown in the embodiments of the present invention provide a method for dynamically moving virtual windows on a virtual reality device. By using depth detection and / or feature point detection, a target block that will not affect the user's operation can be automatically found, and the virtual window can be moved to this target block, thereby reducing occlusion problems and the need for manual operation by the user.

[0145] Although the present invention has disclosed the above-described contents by way of embodiments, it is not intended to limit the present invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the contents defined in the appended claims.

Claims

1. A virtual window configuration device, comprising: The processor is used to generate virtual windows; Depth detection sensors are used to generate depth information based on images; A feature point detection sensor is used to generate feature point information based on the image; as well as A storage device, wherein the processor accesses programs stored in the storage device to implement a depth analysis module and a feature point analysis module, wherein: The depth analysis module is used to analyze the depth information to generate a depth matrix, and to find the depth configuration blocks in the image based on the depth matrix. The feature point analysis module is used to analyze the feature point information to generate a feature point matrix, and to find the feature point configuration blocks in the image based on the feature point matrix. The processor moves the virtual window to the depth configuration block, where the depth value of each pixel in the depth configuration block is greater than a depth threshold value, and the area of ​​the depth configuration block is greater than or equal to a preset block; or it moves the virtual window to the feature point configuration block, where the number of feature points in the feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block.

2. The virtual window configuration device as described in claim 1, further comprising: Gravity sensors are used to detect the orientation of the augmented reality glasses; and A camera is used to take a picture in the direction of the turning position to obtain the image; in, The images contain actual scenes; The processor moves the virtual window to the line-of-sight position; The processor calculates the line-of-sight position of the virtual window in the image based on the turning position.

3. The virtual window configuration device as described in claim 1, wherein, The depth value of each pixel in the depth configuration block is greater than the depth threshold value, and the area of ​​the depth configuration block is greater than or equal to the preset block. The number of feature points in the feature point configuration block is less than the feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block. The processor analyzes the depth configuration block and the feature point configuration block to find overlapping blocks and determines whether the area of ​​the overlapping block is greater than or equal to the preset block. In response to the processor determining that the area of ​​the overlapping block is greater than or equal to the preset block, the processor moves the virtual window to the overlapping block.

4. The virtual window configuration device as described in claim 3, wherein, In response to the processor determining that the area of ​​the overlapping block is not greater than or equal to the preset block, the processor moves the virtual window to the depth configuration block or moves the virtual window to the feature point configuration block.

5. The virtual window configuration device as described in claim 1, wherein, If the processor cannot find a depth configuration block in which the depth value of each pixel in the depth configuration block is greater than a depth threshold value and the area is greater than or equal to a preset block, then the processor searches for a feature point configuration block whose number of feature points is less than a feature point threshold value and whose area is greater than or equal to the preset block. If the processor cannot find a feature point configuration block whose number of feature points is less than the feature point threshold value and whose area is greater than or equal to the preset block, then the processor keeps the virtual window in place.

6. The virtual window configuration device as described in claim 1, wherein, The processor adjusts the virtual window to fit the size of the feature point configuration block or the depth configuration block according to the original aspect ratio of the virtual window, and moves the virtual window to the feature point configuration block.

7. The virtual window configuration device as described in claim 3, wherein, In response to the processor determining that the area of ​​the overlapping block is greater than or equal to the preset block, the processor adjusts the virtual window to fit the size of the overlapping block according to the original aspect ratio of the virtual window, and moves the virtual window to the overlapping block.

8. A method for configuring a virtual window, comprising: Virtual windows are generated by the processor; Depth information is generated from images using depth detection sensors; among which... The processor analyzes the depth information to generate a depth matrix, and uses the depth matrix to identify depth configuration blocks in the image. The feature point detection sensor generates feature point information based on the image, the processor analyzes the feature point information to generate a feature point matrix, and the feature point configuration blocks in the image are found based on the feature point matrix. as well as The processor moves the virtual window to the depth configuration block, where the depth value of each pixel in the depth configuration block is greater than the depth threshold value, and the area of ​​the depth configuration block is greater than or equal to a preset block. Alternatively, the virtual window can be moved to the feature point configuration block, where the number of feature points in the feature point configuration block is less than the feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block.

9. The virtual window configuration method as described in claim 8, further comprising: The orientation of the augmented reality glasses is detected by a gravity sensor; and The image is obtained by taking a picture with a camera facing the turning position; in, The images contain actual scenes; The processor moves the virtual window to the line-of-sight position; The processor calculates the line-of-sight position of the virtual window in the image based on the turning position.

10. The virtual window configuration method as described in claim 8, wherein, The depth value of each pixel in the depth configuration block is greater than the depth threshold value, and the area of ​​the depth configuration block is greater than or equal to the preset block. The number of feature points in the feature point configuration block is less than the feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block. The processor analyzes the depth configuration block and the feature point configuration block to find overlapping blocks and determines whether the area of ​​the overlapping block is greater than or equal to the preset block. In response to the processor determining that the area of ​​the overlapping block is greater than or equal to the preset block, the processor moves the virtual window to the overlapping block.

11. The virtual window configuration method as described in claim 10, further comprising: In response to the processor determining that the area of ​​the overlapping block is not greater than or equal to the preset block, the processor moves the virtual window to the depth configuration block or moves the virtual window to the feature point configuration block.

12. The virtual window configuration method as described in claim 8, further comprising: If a depth configuration block cannot be found where the depth value of each pixel in the depth configuration block is greater than the depth threshold value and the area is greater than or equal to a preset block, then a feature point configuration block with a number of feature points less than the feature point threshold value and an area greater than or equal to the preset block is searched. If a feature point configuration block with a number of feature points less than the feature point threshold value and an area greater than or equal to the preset block is not found, then the virtual window is left in place.

13. The virtual window configuration method as described in claim 8, further comprising: Based on the original aspect ratio of the virtual window, adjust the virtual window to match the size of the feature point configuration block or the depth configuration block; and Move the virtual window to the feature point configuration block.

14. The virtual window configuration method as described in claim 10, wherein, In response to the processor determining that the area of ​​the overlapping block is greater than or equal to the preset block, the processor adjusts the virtual window to fit the size of the overlapping block according to the original aspect ratio of the virtual window, and moves the virtual window to the overlapping block.

15. A virtual window configuration system, comprising: The processor is used to generate virtual windows; and Augmented reality glasses, including: Depth detection sensors are used to generate depth information based on images; as well as A feature point detection sensor is used to generate feature point information based on the image; in, The augmented reality glasses transmit the depth information and the feature point information to the processor; The processor analyzes the depth information to generate a depth matrix and uses the depth matrix to find depth configuration blocks in the image. The processor analyzes the feature point information to generate a feature point matrix, and identifies feature point configuration blocks in the image based on the feature point matrix. The processor moves the virtual window to the depth configuration block, where the depth value of each pixel in the depth configuration block is greater than a depth threshold value, and the area of ​​the depth configuration block is greater than or equal to a preset block; or it moves the virtual window to the feature point configuration block, where the number of feature points in the feature point configuration block is less than a feature point threshold value, and the area of ​​the feature point configuration block is greater than or equal to the preset block.

16. The virtual window configuration system as described in claim 15, wherein, The processor analyzes the depth configuration block and the feature point configuration block to find overlapping blocks and determines whether the area of ​​the overlapping block is greater than or equal to the preset block. In response to the processor determining that the area of ​​the overlapping block is greater than or equal to the preset block, the processor transmits the coordinates of the overlapping block to the augmented reality glasses. In this process, another processor in the augmented reality glasses moves the virtual window to the overlapping block based on the coordinates of the overlapping block.

Images