Virtual reality system and object detection method suitable for virtual reality system

Abstract

An object detection method is suitable for a virtual reality system. The object detection method includes capturing a plurality of first images by a plurality of first cameras of a head-mounted display (HMD). The object detection method also includes capturing a plurality of second images by a plurality of second cameras of a mobile tracker. An object detection model searches for a position of an object in the first images and the second images.

Description

Technical Field

[0001] This invention relates to virtual reality systems, and more particularly to an object detection method suitable for virtual reality systems. Background Technology

[0002] Generally, users need to wear head-mounted displays to experience virtual reality scenes. However, while wearing a head-mounted display, users may become so engrossed in the game that they lose track of their spatial orientation or physical location. If a user needs to find a specific object, such as a phone, tablet, or sneakers, they must remove the head-mounted display, which is quite inconvenient.

[0003] Therefore, how to enable users to access physical objects without removing the head-mounted display device has become one of the problems to be solved in this field. Summary of the Invention

[0004] This invention provides a virtual reality system, including a head-mounted display (HMD) and a tracker. The HMD includes multiple first cameras, a storage device, and a processor. The first cameras capture multiple first images. The processor accesses an object detector stored in the storage device and executes the object detector, wherein the processor performs the following: inputting the first images and their respective first map location data into the object detector. The tracker includes multiple second cameras. The second cameras capture multiple second images. The tracker transmits the respective second map location data of the second images to the object detector, and the object detector updates an environmental overlay image based on the first images, the first map location data, the second images, and the second map location data, and searches for the location of an object in the first and second images.

[0005] This invention provides an object detection method applicable to virtual reality systems. The object detection method includes capturing multiple first images using multiple first cameras in a head-mounted display (HMD). A processor in the HMD executes the following: inputting the first images and their respective first map location data into an object detection model. Multiple second images are captured using multiple second cameras in a tracker. The tracker transmits the respective second map location data of the second images to the object detection model. The object detection model updates an environmental overlay image based on the first images, the first map location data, the second images, and the second map location data, and searches for the location of an object within the first and second images.

[0006] In summary, the embodiments of the present invention provide a virtual reality system and an object detection method applicable to the virtual reality system. Through the object detection model, the position of an object can be found from the images captured by the head-mounted display device and the motion locator. This allows the user to indicate the position of physical objects in the environment through the head-mounted display device while wearing it, thereby enabling the user to reach physical objects without removing the head-mounted display device. This makes the virtual reality system more convenient and provides a better user experience. Attached Figure Description

[0007] Figure 1A This is a schematic diagram of a virtual reality system according to an embodiment of the present invention; Figure 1B This is a schematic diagram of an object detection model according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating an object detection method according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating an object detection method according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating an object detection method according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating an object detection method according to an embodiment of the present invention.

[0008] Symbol explanation: 10: Head-mounted display device 20: Object Detection Model 30: Mobile Positioner S1~C7: Steps IMG: Images RST: Image Output Results class 0: Classification code F1: Specific HMD field of view MP: Feature Points USR: User BX: Selected area F2: Object field of view F3: Field of view for a specific mobile locator F4: 3D Intersection Range OB: Object position Detailed Implementation

[0009] The following description illustrates a preferred embodiment of the invention and is intended to describe the basic spirit of the invention, but is not intended to limit the invention. The actual scope of the invention must be determined by referring to the claims that follow.

[0010] It must be understood that the words “comprising” and “including” used in this specification are used to indicate the presence of specific technical features, values, method steps, work processes, elements and / or components, but do not preclude the addition of more technical features, values, method steps, work processes, elements, components, or any combination thereof.

[0011] The use of terms such as "first," "second," and "third" in claims to modify elements in the claims is not intended to indicate a priority order, a prior relationship, or that one element precedes another, or the chronological order of the execution of method steps; it is only used to distinguish elements with the same name.

[0012] Please refer to Figure 1A , Figure 1A This is a schematic diagram illustrating a virtual reality system 100 according to an embodiment of the present invention. In one embodiment, the virtual reality system 100 includes a head-mounted display (HMD) 10 and a tracker 30. In one embodiment, the head-mounted display 10 includes a plurality of first cameras, a storage device, and a processor. In one embodiment, the tracker 30 includes a plurality of second cameras.

[0013] In one embodiment, the storage device in the head-mounted display device 10 may be implemented as a read-only memory, flash memory, floppy disk, hard disk, optical disk, USB flash drive, magnetic tape, a database accessible by a network, or a storage medium with the same function that a technician can easily conceive of.

[0014] In one embodiment, the processor in the head-mounted display device 10 may be implemented by a microcontroller, a microprocessor, a digital signal processor, an application-specific integrated circuit (ASIC), or a logic circuit.

[0015] In one embodiment, a plurality of first cameras in the head-mounted display device 10 are disposed at different positions in the head-mounted display device 10 and can capture environmental images.

[0016] In one embodiment, the processor in the head-mounted display device 10 is used to access an object detector 20 stored in a storage device to execute the object detector model.

[0017] In one embodiment, the object detection model 20 may be a thread that is accessed from and executed by a processor in the head-mounted display device 10 from a storage device.

[0018] Please see Figure 1B , Figure 1B This is a schematic diagram illustrating an object detection model 20 according to an embodiment of the present invention. In one embodiment, the object detection model 20 can be implemented by a region-based convolutional neural network (R-CNN), whose network function is as follows: Figure 1B As shown, a convolutional layer can be roughly divided into an encoder and a decoder to select a specific object within an image. When an image (e.g., an IMG image) is input to the encoder of the convolutional layer, after neural network-like computation, the decoder outputs a box head and a class head. In this example, a table lamp is the default specific object. Using the information from the box head (used to find and select the specific object) and the class head (used to classify the object), the table lamp region in the IMG image can be selected, i.e., an image output result RST containing the selected region is generated. The image output result RST contains the selected table lamp region and classification information (e.g., the classification code class 0 corresponding to the table lamp). Thus, the object detection model 20 can select and classify specific objects in an image.

[0019] In one embodiment, the object detection model 20 is trained on images of multiple pre-set objects, such as mobile phones, tablets, mice, sneakers, etc., before the virtual reality system 100 leaves the factory. Training is considered complete when the training results converge, and the trained object detection model 20 is stored in the storage device of the head-mounted display device 10. However, this application is not limited to this; the object detection model 20 can also be updated or trained after leaving the factory upon user request.

[0020] In one embodiment, the trained object detection model 20 can also be burned into a chip to become a piece of hardware in the head-mounted display device 10.

[0021] In one embodiment, the object detection model 20 can also be implemented by a hardware circuit.

[0022] In one embodiment, after the virtual reality system 100 is manufactured, the user can select the location of the object to be detected through a list of detected objects displayed on the head-mounted display device 10. For example, the user can select objects such as mobile phones and sneakers. When the virtual reality system 100 is running, it can detect the location of the object selected by the user through the object detection model 20.

[0023] In one embodiment, the motion locator 30 includes a plurality of second cameras. For example, the motion locator 30 can be strapped to the user's wrist. In virtual reality, the virtual reality system 100 can use the motion locator 30 to know the position and movement status of the wrist in virtual reality.

[0024] In one embodiment, the mobile locator 30 has multiple second cameras that shoot in different directions, thereby capturing environmental images from multiple perspectives. For ease of description, the images captured by the second cameras will be referred to as second images below.

[0025] In one embodiment, the virtual reality system 100 includes a plurality of motion locators 30, which can be fixed to the user's limbs, racket, or other objects.

[0026] In one embodiment, the head-mounted display device 10 and the motion locator 30 can establish a communication connection via wired or wireless means.

[0027] The following statement is made through Figure 1A The virtual reality system 100 includes a head-mounted display device 10, an object detection model 20, and a motion locator 30 to implement the object detection method. The location of the object to be found in steps S1-S7 is exemplified by the location of a mobile phone, but the invention is not limited thereto.

[0028] In step S1, the processor of the head-mounted display device 10 inputs these first images and a first map location data of each of these first images into the object detection model 20.

[0029] In step S2, the mobile locator 30 transmits the second map location data of each of these second images to the object detection model 20.

[0030] In one embodiment, these second images are captured by multiple cameras on the mobile locator 30.

[0031] In step S3, the object detection model 20 updates an environmental overlay image based on the first images (frames), the first map location data, the second images, and the second map location data.

[0032] In one embodiment, when the object detection model 20 determines that the first map location data and the second map location data are different from the existing data of the same location, it means that the actual environment may have been changed, such as a family member moving the placement of the mobile phone. Therefore, the object detection model 20 updates the environmental overlay image to overlay the existing corresponding map location data and image with the map location data and image obtained at the latest time.

[0033] In one embodiment, the cameras of the head-mounted display device 10 and the motion locator 30 only capture key information, such as areas containing map points (MPs). In another embodiment, the head-mounted display device 10 and the motion locator 30 ignore and do not process previously transmitted images of the same location that have not changed.

[0034] In this context, map points (MPs) can refer to feature points. Areas in an image that have distinct edges, corners, color blocks, unique colors, or are easily identifiable can be considered feature points. Feature points are also marked on regions of interest (ROIs), which are, for example, the locations of objects that the user selects to find (or display).

[0035] Since map point MP can be regarded as a feature point, the following explanation uses feature point MP.

[0036] In one embodiment, the environment overlay image must include all feature points MP.

[0037] In one embodiment, the object detection model 20 uses minimal imagery to cover the surrounding environment, and these images must contain all feature points (MPs). In other words, each image transmitted to the object detection model 20 is a keyframe, and the keyframe must cover a certain extent of the environment map points (MPs) to be retained by the object detection model 20.

[0038] Therefore, the object detection model 20 can cover the surrounding environment with the minimum number of images. Before transmitting images, both the head-mounted display device 10 and the motion locator 30 check whether images of the same location have already been transmitted, whether the images overlap, or whether the images have been modified. In other words, the head-mounted display device 10 and the motion locator 30 only transmit key images—that is, images that have not been transmitted before or updated images (e.g., images of the same location where objects have been moved)—to the object detection model 20. Thus, the object detection model 20 will not receive duplicate images.

[0039] In one embodiment, the total number of feature points (MPs) is determined by the Simultaneous Localization and Mapping (SLAM) technique, which continuously builds map points (MPs) until no more need to be added, indicating that the approximately important feature points (MPs) of this environment have been found.

[0040] In one embodiment, the object detection model 20 can set a threshold value to determine whether to retain the received key image. For example, a threshold value of 15% means that the map point MP in the key image must share less than 15% of the map point MPs with existing key images to be retained. For example, if the currently received key image has a total of 100 map point MPs, and among the key images already retained by the object detection model 20, any key image contains more than 16 shared map points, then the object detection model 20 will not retain this key image.

[0041] In one embodiment, the object detection model 20 retains key images that have a viewpoint for each map point MP. For example, key images A and key images B can jointly observe a map point MP. If the viewpoint between key images A and key images B is greater than 60 degrees, then this map point MP will regard key images A and key images B as different map points.

[0042] In one embodiment, when a new key image with a similar location (e.g., a distance of less than 5 cm) contains 87 different map points (MP), which is more than the 86 map points (MP) contained in the current key image, the object detection model 20 discards the current key image and retains the new key image.

[0043] Therefore, when the map point MP changes significantly (which may be due to changes in the actual environment, such as someone cleaning, moving objects, or the user moving to another room), the object detection model 20 will update the key images of the map point MP that have been changed at any time.

[0044] In step S4, the object detection model 20 searches for the location of an object in the first image and the second image.

[0045] In one embodiment, the virtual reality system 100 can receive a user's setting to locate a mobile phone (object). For example, the user selects the desired location of the mobile phone from the list of detectable objects displayed on the head-mounted display device 10's screen. The object detection model 20 then searches for the mobile phone's location in the first and second images. Since the object detection model 20 is a pre-trained model, the appearance of the mobile phone can be identified from the first and second images.

[0046] Thus, the user can obtain the location of the mobile phone. More specifically, in step S5, after the object detection model 20 finds the location of the mobile phone, the object detection model 20 transmits the location of the mobile phone (e.g., three-dimensional coordinates) to the head-mounted display device 10.

[0047] Thus, the display of the head-mounted display device 10 can show the location of the mobile phone to the user, allowing the user to pick up the mobile phone without having to remove the head-mounted display device 10.

[0048] In one embodiment, the display of the head-mounted display device 10 shows the phone's location in a mode that allows users to view the surrounding real-world environment, such as VR office or see-through mode, without affecting the user's gaming experience.

[0049] In one embodiment, the object detection method may include steps S6 to S7.

[0050] In step S6, the processor in the head-mounted display device 10 determines whether the virtual reality system 100 is currently in a mode that refers to the surrounding real situation. If yes, it proceeds to step S7; otherwise, it is not enabled.

[0051] In step S7, the display of the head-mounted display device 10 displays the position of the object (in this example, the position of the object refers to the position of the mobile phone).

[0052] In one embodiment, the head-mounted display device 10 transmits initial map information to the motion locator 30 during initialization. Both the head-mounted display device 10 and the motion locator 30 use the same Simultaneous Localization and Mapping (SLAM) technology. The motion locator 30 integrates its own collected map information based on the initial map information transmitted from the head-mounted display device 10 to improve its own initial map information. The motion locator 30 periodically transmits poses to the head-mounted display device 10. The head-mounted display device 10 uses these poses to calibrate the pose of the motion locator 30 as perceived by the head-mounted display device 10.

[0053] In one embodiment, when the relative postures of both the head-mounted display device 10 and the motion locator 30 have been corrected to a stable state, the processor of the head-mounted display device 10 begins to focus on detecting the posture of the head-mounted display device 10.

[0054] In one embodiment, multiple feature points MP can be used during positioning to make the positioning of the head-mounted display device 10 and the motion locator 30 more accurate using synchronous positioning and mapping techniques.

[0055] Please see Figures 2-5 , Figures 2-5 This is a schematic diagram illustrating an object detection method according to an embodiment of the present invention. In one embodiment, please refer to Figure 2 When the user USR wears the head-mounted display device 10 and the motion locator 30, multiple first cameras on the head-mounted display device 10 capture multiple first images.

[0056] In one embodiment, these first images include a specific HMD image that covers a specific HMD field of view (FOV) F1, which contains a plurality of feature points MP (each feature point MP is represented by a solid dot).

[0057] In one embodiment, the object detection model 20 detects whether an object image (e.g., a sneaker) is contained within a specific HMD field of view F1. In response to the object detection model 20 detecting that an object image is contained within the specific HMD field of view F1, the object detection model 20 selects an object field of view F2 of the object image from the specific HMD field of view F1; wherein the object field of view F2 is smaller than the specific HMD field of view F1.

[0058] In one embodiment, please refer to Figure 3 When the user USR puts on the head-mounted display device 10, at least one of the first cameras captures an image of an object (e.g., a sneaker). In response to the object detection model 20 detecting that the object image is contained in a specific HMD field of view F1, the object detection model 20 selects an object field of view F2 of the object image from the specific HMD field of view F1.

[0059] Depend on Figure 3 It can be seen that the field of view of a specific HMD field of view F1 is larger than that of the object field of view F2, and the object field of view F2 is located within the specific HMD field of view F1.

[0060] Depend on Figure 3As can be seen, after the object detection model 20 detects the feature points MP of the actual sneaker, it selects the sneaker and forms a bounding box BX, which can be three-dimensional. Therefore, the object detection model 20 can select the object field of view F2 of the object image along the projection range of this bounding box BX.

[0061] In this context, feature point MP within the selected area BX is the same as feature point MP within the object's field of view F2. The overlapping rectangular blocks of varying sizes within the object's field of view F2 represent the sneakers. The selected area BX is located within the object's field of view F2, meaning the sneakers are selected. For ease of explanation, the actual appearance of the sneakers within the selected area BX, which is the image actually seen by the user (USR), is shown in [the image / description]. Figure 3 It is shown on the right.

[0062] In one embodiment, the object field of view F2 and the specific HMD field of view F1 are each a cone-shaped three-dimensional spatial image projected from a specific HMD camera among the same set of first cameras. In other words, after the object detection model 20 selects the position of the sneaker and generates a selection range BX, the processor of the head-mounted display device 10 or the object detection model 20 can capture the light projection path along the selection range BX captured by this specific HMD camera to obtain a cone-shaped three-dimensional spatial image (i.e., the object field of view F2 of the object image).

[0063] Please see Figure 4 , Figure 4 This is a scene diagram shown from the perspective of the user's USR (Underwriter's Relationship to the User). Figure 3 This is a scene diagram based on the user's USR side view angle. Figure 4 and Figure 3 This is merely to illustrate a different perspective to explain this example. In one embodiment, the object detection model 20 sets an initial three-dimensional object position (i.e., the selected area BX) based on multiple specific feature points within the object's field of view F2 (i.e., five specific feature points within the selected area BX). Since these five specific feature points are all located in three-dimensional space, the initial three-dimensional object position connected by these five specific feature points will be a three-dimensional space.

[0064] In one embodiment, please refer to Figure 5 Among the second images captured by the second camera on the locator 30, there is a specific locator image that covers a specific locator field of view F3. The specific locator field of view F3 and the object field of view F2 have a three-dimensional intersection range F4.

[0065] In one embodiment, the field of view F3 of a specific mobile locator is also a cone-shaped three-dimensional spatial image.

[0066] In one embodiment, the object detection model 20 sets the object position OB based on multiple target feature points MP in the three-dimensional intersection range F4.

[0067] In one embodiment, the object detection model 20 regards the feature point MP in the three-dimensional intersection range F4 as the target feature point MP, which represents the feature point MP to which the user USR belongs.

[0068] Depend on Figure 5 As can be seen from the example, there are two target feature points MP in the three-dimensional intersection range F4. The object detection model 20 sets the object position OB based on these two target feature points MP (as in the previous example, this is the position where the sneakers are placed).

[0069] In one embodiment, if the user USR initially sets the location to find the phone, the object detection model 20 performs the above-described corresponding... Figures 2-4 After the described steps, as follows Figure 5 As shown, the range of these two target feature points MP is considered as the location of the mobile phone. In one embodiment, when the display of the head-mounted display device 10 is in a mode that allows users to view the surrounding real-world situation, such as VR office or see-through mode, the display of the head-mounted display device 10 shows the location of the mobile phone, allowing the user to access the mobile phone without removing the head-mounted display device 10, and without affecting the user's gaming experience.

[0070] In one embodiment, even if the object detection model 20 receives a new first image and / or a second image updated within a specific HMD field of view F1, thereby generating a new three-dimensional intersection range F4, if the number of feature points MP in the space corresponding to the previous object position OB (for example, if there are 2 target feature points MP in the previous object position OB and 5 target feature points MP in the new three-dimensional intersection range F4, indicating that the object position OB has more target feature points MP), the number of feature points MP in the new three-dimensional intersection range F4 does not decrease, which means that the previous object position OB is still correct and the previous object position OB is still selected.

[0071] In summary, the embodiments of the present invention provide a virtual reality system and an object detection method applicable to the virtual reality system. Through the object detection model, the position of an object can be found from the images captured by the head-mounted display device and the motion locator. This allows the user to indicate the position of physical objects in the environment through the head-mounted display device while wearing it, thereby enabling the user to reach physical objects without removing the head-mounted display device. This makes the virtual reality system more convenient and provides a better user experience.

[0072] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the scope of the invention. Any person skilled in the art can make some modifications without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A virtual reality system, comprising: A head-mounted display device, fixed to the user's head, includes: Multiple first cameras are used to capture environmental images, and multiple first images are captured, the environment including an object; A storage device; as well as A processor is configured to access an object detection model stored in the storage device, and to execute the object detection model, wherein the processor performs: The multiple first images and their respective first map location data are input into the object detection model; and A mobile locator, fixed to the user's limb, includes: Multiple second cameras are used to capture images of the environment and to capture multiple second images; The mobile locator transmits multiple second images and their respective second map location data to the object detection model. The object detection model updates an environmental overlay image based on the multiple first images, the multiple first map location data, the multiple second images, and the multiple second map location data. When it is determined that the multiple first map location data and the multiple second map location data are different from existing data for the same location, the object detection model updates the environmental overlay image. The object detection model determines the three-dimensional intersection range of the first image and the second image, which includes multiple target feature points corresponding to the object, and searches for the location of an object in the first image and the second image based on the multiple target feature points.

2. The virtual reality system as described in claim 1, wherein, The plurality of first images include a specific HMD image, which covers a specific HMD field of view, and the specific HMD field of view contains multiple feature points. The specific HMD image is an image captured by a specific HMD camera among the plurality of first cameras.

3. The virtual reality system of claim 2, wherein the object detection model detects whether an object image is contained within the specific HMD field of view, and in response to the object detection model detecting that the object image is contained within the specific HMD field of view, the object detection model selects an object field of view of the object image from the specific HMD field of view. in, The field of view of the object is smaller than the field of view of the specific HMD.

4. The virtual reality system of claim 3, wherein the object field of view and the specific HMD field of view are each a cone-shaped three-dimensional spatial image projected from a specific HMD camera among the plurality of first cameras.

5. The virtual reality system of claim 3, wherein the object detection model sets an initial three-dimensional object position based on multiple specific feature points within the object's field of view.

6. The virtual reality system of claim 5, wherein the plurality of second images includes a specific motion locator image, the specific motion locator image covering the field of view of the specific motion locator: in, The field of view of the specific mobile locator and the field of view of the object have a three-dimensional intersection range.

7. The virtual reality system of claim 6, wherein the object detection model sets the object position based on multiple target feature points in the three-dimensional intersection range.

8. An object detection method, applicable to a virtual reality system, comprising: Multiple first cameras of a head-mounted display device are used to capture environmental images, and multiple first images are captured. The head-mounted display device is fixed to the user's head, and the environment includes an object. The multiple first images and their respective first map location data are input into an object detection model; and Multiple second cameras in a mobile locator are used to capture images of the environment and take multiple second images, the mobile locator being fixed to the user's limbs. The mobile locator transmits the multiple second images and their respective second map location data to the object detection model. The object detection model updates an environmental overlay image based on the multiple first images, the multiple first map location data, the multiple second images, and the multiple second map location data. When it is determined that the multiple first map location data and the multiple second map location data are different from existing data for the same location, the object detection model updates the environmental overlay image. The object detection model determines the three-dimensional intersection range of the first image and the second image, which includes multiple target feature points corresponding to the object, and searches for the location of an object in the first image and the second image based on the multiple target feature points.

9. The object detection method as described in claim 8, wherein, The plurality of first images include a specific HMD image, which covers a specific HMD field of view, and the specific HMD field of view contains multiple feature points. The specific HMD image is an image captured by a specific HMD camera among the plurality of first cameras.

10. The object detection method as described in claim 9, further comprising: The object detection model detects whether an object image is contained within the specific HMD field of view. In response to the object detection model detecting that the object image is contained within the specific HMD field of view, the object detection model selects an object field of view of the object image from the specific HMD field of view. in, The field of view of the object is smaller than the field of view of the specific HMD.

11. The object detection method of claim 10, wherein the object field of view and the specific HMD field of view are each a cone-shaped three-dimensional spatial image projected from a specific HMD camera among the plurality of first cameras.

12. The object detection method as described in claim 10, wherein the object detection model sets an initial three-dimensional object position based on multiple specific feature points in the object's field of view.

13. The object detection method as described in claim 12, wherein the plurality of second images includes a specific motion locator image, the specific motion locator image covering the field of view of the specific motion locator: in, The field of view of the specific mobile locator and the field of view of the object have a three-dimensional intersection range.

14. The object detection method as described in claim 13, further comprising: The object detection model sets the object position based on multiple target feature points within the three-dimensional intersection range.

Images