Virtual reality device and handle positioning method

Abstract

The present application provides an embodiment, belonging to virtual reality technology, and provides a virtual reality device and a handle positioning method, the virtual reality device includes a display; a camera, configured to collect multiple frames of images of a handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light; a controller connected to the camera, the controller is configured to: extract at least one light spot from the multiple frames of images of the handle; encode the at least one light spot to form a first coding information; remove the interference light spot in the at least one light spot according to the first coding information and the second coding information of the at least one indicator light, and determine the light spot corresponding to the at least one indicator light; determine the position of the handle according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images. Compared with the prior art, the embodiment of the present application can remove the interference light spot by the coding information preset by the indicator light when positioning the handle, thereby improving the accuracy of the handle positioning.

Description

Technical Field

[0001] The embodiments of the present application relate to virtual reality technology, and more specifically, to a virtual reality device and a handle positioning method. Background Art

[0002] At present, virtual reality (VR) devices are widely used in education and training, fire drills, virtual driving, real estate and other fields, which can bring users an immersive visual feast. When using virtual reality devices, the movement trajectory of the handle connected to the VR device can usually be used to determine the user's instructions. Therefore, during the use of virtual reality devices, the handle needs to be positioned.

[0003] In the prior art, during the positioning of the handle, the indicator light on the handle will flash. At the same time, the camera on the VR device will capture the image of the handle, extract and encode the light spot formed by the flashing indicator light in the image, and determine the light spot corresponding to the indicator light based on the encoding information. Finally, the VR device completes the positioning of the handle based on the position of the light spot corresponding to the indicator light in the image.

[0004] However, in a real environment, there may be other light sources around the handle, which may form interfering light spots on the image of the handle and affect the accuracy of handle positioning. Summary of the invention

[0005] The exemplary embodiments of the present application provide a virtual reality device and a handle positioning method, which can improve the accuracy of handle positioning.

[0006] In a first aspect, an embodiment of the present application provides a virtual reality device, including:

[0007] monitor;

[0008] A camera is configured to collect multiple frames of images of a handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light;

[0009] A controller connected to the camera, the controller being configured to:

[0010] Extracting at least one light spot from the multiple frames of images of the handle;

[0011] Encoding the at least one light spot to form first coded information;

[0012] According to the first coding information and the second coding information of the at least one indicator light, removing the interfering light spot in the at least one light spot, and determining the light spot corresponding to the at least one indicator light;

[0013] The position of the handle is determined according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images.

[0014] In some embodiments of the present application, the controller is specifically configured as follows:

[0015] Determining brightness changes of the at least one light spot in the multiple frames of images according to changes in diameter of the at least one light spot in the multiple frames of images;

[0016] According to the brightness and darkness changes in the multiple frames of images, the at least one light spot is encoded to form first encoding information.

[0017] In some embodiments of the present application, the controller is specifically configured as follows:

[0018] According to the brightness changes in the multiple frames of images, and the corresponding relationship between the brightness and darkness of the light spot and the bitmap information, the at least one light spot is encoded to form first encoding information.

[0019] In some embodiments of the present application, the controller is specifically configured as follows:

[0020] If there is second coding information that is the same as the first coding information, determining that the light spot corresponding to the first coding information is the light spot corresponding to the at least one indicator light;

[0021] If there is no second coding information identical to the first coding information, determining that the light spot corresponding to the first coding information is an interference light spot;

[0022] The interfering light spot is removed.

[0023] In some embodiments of the present application, the controller is further configured to:

[0024] Sending synchronization information to the handle, wherein the synchronization information includes a flashing cycle of the at least one indicator light.

[0025] In a second aspect, an embodiment of the present application provides a handle positioning method, which is applied to a virtual reality device, wherein the virtual reality device is connected to the handle, and the method includes:

[0026] Collecting multiple frames of images of the handle, wherein the handle is provided with at least one indicator light;

[0027] Extracting at least one light spot from the multiple frames of images of the handle;

[0028] Encoding the at least one light spot to form first coded information;

[0029] According to the first coding information and the second coding information of the at least one indicator light, removing the interfering light spot in the at least one light spot, and determining the light spot corresponding to the at least one indicator light;

[0030] The position of the handle is determined according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images.

[0031] In some embodiments of the present application, encoding the at least one light spot to form first coded information includes:

[0032] Determining brightness changes of the at least one light spot in the multiple frames of images according to changes in diameter of the at least one light spot in the multiple frames of images;

[0033] According to the brightness and darkness changes in the multiple frames of images, the at least one light spot is encoded to form first encoding information.

[0034] In some embodiments of the present application, encoding the at least one light spot according to the brightness changes in the multiple frames of images to form first coded information includes:

[0035] According to the brightness changes in the multiple frames of images, and the corresponding relationship between the brightness and darkness of the light spot and the bitmap information, the at least one light spot is encoded to form first encoding information.

[0036] In some embodiments of the present application, removing the interfering light spot in the at least one light spot according to the first coding information and the second coding information of the at least one indicator light, and determining the light spot corresponding to the at least one indicator light, includes:

[0037] If there is second coding information that is the same as the first coding information, determining that the light spot corresponding to the first coding information is the light spot corresponding to the at least one indicator light;

[0038] If there is no second coding information identical to the first coding information, determining that the light spot corresponding to the first coding information is an interference light spot;

[0039] The interfering light spot is removed.

[0040] In some embodiments of the present application, before acquiring the multiple frames of images of the handle, the method further includes:

[0041] Sending synchronization information to the handle, wherein the synchronization information includes a flashing cycle of the at least one indicator light.

[0042] In a third aspect, the present application further provides an electronic device, including:

[0043] Memory and processor;

[0044] The memory is used to store executable instructions of the processor;

[0045] The processor is configured to perform the method of the second aspect by executing the executable instructions.

[0046] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, and when the computer program is executed by a processor, any possible method in the second aspect is implemented.

[0047] The virtual reality device and handle positioning method provided by the embodiment of the present application include a display; a camera, which is configured to collect multiple frames of images of a handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light; a controller connected to the camera, which is configured to: extract at least one light spot from the multiple frames of images of the handle; encode the at least one light spot to form first coding information; remove interference light spots in the at least one light spot according to the first coding information and the second coding information of the at least one indicator light, and determine the light spot corresponding to the at least one indicator light; determine the position of the handle according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images. Compared with the prior art, the embodiment of the present application can remove interference light spots through coding information when positioning the handle, thereby improving the accuracy of the handle positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0049] Figure 1 Schematic diagrams of operation scenarios between a VR device and a handle according to some embodiments are exemplarily shown in FIG.

[0050] Figure 2 exemplarily shows a hardware configuration block diagram of a VR device according to some embodiments;

[0051] Figure 3 The configuration block diagram of the handle according to some embodiments is exemplarily shown;

[0052] Figure 4 Schematic diagram of software configuration in a VR device according to some embodiments is exemplarily shown in FIG.

[0053] Figure 5 exemplarily shows a schematic diagram of an icon control interface display of an application in a VR device according to some embodiments;

[0054] Figure 6 A schematic diagram of a handle positioning process according to some embodiments is exemplarily shown;

[0055] Figure 7a-7c A schematic diagram of arrangement of indicator lights according to some embodiments is exemplarily shown;

[0056] Figure 8a-8b A schematic diagram exemplarily illustrates a bright spot according to some embodiments;

[0057] Fig. 9 A schematic diagram of another process of positioning a handle according to some embodiments is exemplarily shown;

[0058] Fig.10 A schematic structural diagram of a virtual reality device according to some embodiments is exemplified. DETAILED DESCRIPTION

[0059] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0060] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0061] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0062] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0063] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0064] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0065] Figure 1 FIG. 2 is a schematic diagram showing an operation scenario between a VR device and a handle according to some embodiments. Figure 1 As shown in FIG. 1 , a user can operate the VR device 200 through the handle 100 .

[0066] In some embodiments, the handle 100 can also become a controller, and the communication between the handle 100 and the VR device 200 includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, etc., to control the VR device 200 wirelessly or in a wired manner. The user can input user commands through buttons, voice input, gesture input, etc. on the VR device 200 to control the VR device 200. For example, the user can control the VR device 200 through the volume plus and minus keys, voice input keys, menu keys, power on and off keys, etc. on the handle 100, or by moving the handle 100 to make corresponding gestures and input corresponding control commands.

[0067] In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices may also be used to control the VR device 200. For example, the VR device 200 may be controlled using an application running on the smart device.

[0068] like Figure 1 It is also shown that the VR device 200 also communicates data with the server 300 through a variety of communication methods. The VR device 200 may be allowed to communicate and connect via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 300 may provide various content and interactions to the VR device 200. For example, the VR device 200 sends and receives information, interacts with an electronic program guide (EPG), receives software program updates, or accesses a remotely stored digital media library. The server 300 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service content such as game operation, video on demand, and advertising services are provided through the server 300.

[0069] The VR device 200 may be a VR helmet, VR glasses, etc. The type, size, and resolution of the specific VR device 200 are not limited, and those skilled in the art will appreciate that the VR device 200 may be modified in performance and configuration as needed.

[0070] Figure 2 Schematic diagram of the hardware configuration of a VR device according to some embodiments is shown in FIG.

[0071] In some embodiments, the VR device 200 includes at least one of a controller 250, a tuner and demodulator 210, a communicator 220, a detector 230, an input / output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, a user interface 265, and an external device interface 240.

[0072] In some embodiments, the display 275 is a component for receiving the image signal outputted from the first processor and displaying video content and images as well as a menu control interface.

[0073] In some embodiments, the display 275 includes a display screen component for presenting images and a driving component for driving image display.

[0074] In some embodiments, the video content may be displayed by various broadcast signals received through wired or wireless communication protocols, or various image contents sent from a network server may be displayed by receiving a network communication protocol.

[0075] In some embodiments, the display 275 is used to present a user manipulation UI interface generated in the display device 200 and used to control the display device 200 .

[0076] In some embodiments, depending on the type of display 275, a driving component for driving the display is also included.

[0077] In some embodiments, the communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

[0078] In some embodiments, the display device 200 can establish control signal and data signal transmission and reception with the handle 100 or the content providing device through the communicator 220 .

[0079] In some embodiments, the user interface 265 may be used to receive infrared control signals from the handle 100 (eg, an infrared remote controller, etc.).

[0080] In some embodiments, the detector 230 is a signal used by the VR device 200 to collect external environment or interact with the outside.

[0081] In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes by collecting ambient light.

[0082] In some embodiments, the detector 230 may also include an image collector, such as a camera, a camera, etc., which can be used to collect external environment scenes, as well as to collect user attributes or gestures for interacting with the user. It can adaptively change display parameters and recognize user gestures to achieve the function of interacting with the user.

[0083] In some embodiments, the detector 230 may also include a temperature sensor or the like, such as by sensing the ambient temperature.

[0084] In some embodiments, the VR device 200 can adaptively adjust the display color temperature of the image. For example, when the temperature is high, the VR device 200 can be adjusted to display a cooler color temperature of the image, or when the temperature is low, the VR device 200 can be adjusted to display a warmer color temperature of the image.

[0085] In some embodiments, the detector 230 may also be a sound collector, such as a microphone, which can be used to receive the user's voice, for example, including a voice signal of a control instruction of the user controlling the VR device 200, or collect environmental sounds to identify the type of environmental scene, so that the VR device 200 can adaptively adapt to the environmental noise.

[0086] In some embodiments, Figure 2 As shown, the input / output interface 255 is configured to perform data transmission between the controller 250 and other external devices or other controllers 250, such as receiving video signal data and audio signal data, or command instruction data from external devices.

[0087] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as a high-definition multimedia interface HDMI interface, an analog or digital high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. It is also possible that the above multiple interfaces form a composite input / output interface.

[0088] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 can control the overall operation of the VR device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 can perform operations related to the object selected by the user command.

[0089] In some embodiments, the object may be any of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object may include, for example, displaying operations such as a page, document, or image connected to the hyperlink, or executing operations corresponding to a program of the icon. User commands for selecting UI objects may be commands inputted through various input devices (e.g., a mouse, a keyboard, a touchpad, etc.) connected to the VR device 200, or voice commands corresponding to voices spoken by the user.

[0090] like Figure 2 As shown, the controller 250 includes at least one of a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor 270, an audio processor 280, other processors 253 (for example: a graphics processor (Graphics Processing Unit, GPU), a central processing unit 254 (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus). Among them, the communication bus connects various components.

[0091] In some embodiments, RAM 251 is used to store temporary data for the operating system or other running programs.

[0092] In some embodiments, ROM 252 is used to store various system startup instructions.

[0093] In some embodiments, ROM 252 is used to store a basic input / output system, called Basic Input Output System (BIOS), which is used to complete the power-on self-test of the system, the initialization of each functional module in the system, the basic input / output driver of the system, and the booting operating system.

[0094] In some embodiments, upon receiving a power-on signal, the VR device 200 starts to power on, and the CPU runs the system startup instruction in the ROM 252, and copies the temporary data of the operating system stored in the memory to the RAM 251, so as to start or run the operating system. After the operating system is started, the CPU copies the temporary data of various application programs in the memory to the RAM 251, so as to start or run various application programs.

[0095] In some embodiments, the CPU processor 254 is used to execute operating system and application instructions stored in the memory, and to execute various applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video contents.

[0096] In some exemplary embodiments, the CPU processor 254 may include multiple processors. The multiple processors may include a main processor and one or more sub-processors. The main processor is used to perform some operations of the VR device 200 in the pre-power-on mode and / or display the screen in the normal mode. One or more sub-processors are used for an operation in the standby mode and the like.

[0097] In some embodiments, the graphics processor 253 is used to generate various graphic objects, such as icons, operation menus, and user input command display graphics, etc. It includes an operator that performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes. It also includes a renderer that renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

[0098] In some embodiments, the video processor 270 is configured to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal, and obtain a signal that can be directly displayed or played on the VR device 200.

[0099] In some embodiments, the video processor 270 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc.

[0100] Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module demultiplexes it into video signals and audio signals.

[0101] The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

[0102] The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the scaled video image to generate an image signal for display.

[0103] The frame rate conversion module is used to convert the input video frame rate, such as converting the 60Hz frame rate to 120Hz frame rate or 240Hz frame rate. The usual format is implemented by interpolation.

[0104] The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting RGB data signal.

[0105] In some embodiments, the graphics processor 253 can be integrated with the video processor or can be separately configured. When integrated, it can perform processing of graphic signals output to the display. When separately configured, it can perform different functions respectively, such as a GPU+FRC (Frame Rate Conversion) architecture.

[0106] In some embodiments, the audio processor 280 is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to the standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

[0107] In some embodiments, the video processor 270 may include one or more chips. The audio processor may also include one or more chips.

[0108] In some embodiments, the video processor 270 and the audio processor 280 may be separate chips, or may be integrated with the controller into one or more chips.

[0109] In some embodiments, the audio output receives the sound signal output by the audio processor 280 under the control of the controller 250, such as the speaker 286, and in addition to the speaker carried by the VR device 200 itself, can be output to the external audio output terminal of the generating device of the external device, such as an external audio interface or a headphone interface, etc., and can also include a short-range communication module in the communication interface, for example, a Bluetooth module for Bluetooth speaker sound output.

[0110] The power supply 290, under the control of the controller 250, uses the power input from the external power supply to provide power supply support for the VR device 200. The power supply 290 may include a built-in power supply circuit installed inside the VR device 200, or may be an external power supply installed on the VR device 200 to provide a power supply interface for an external power supply in the VR device 200.

[0111] The user interface 265 is used to receive a user input signal, and then send the received user input signal to the controller 250. The user input signal may be a remote control signal received by an infrared receiver, and may receive various user control signals through a network communication module.

[0112] In some embodiments, the user inputs a user command through the handle 100 , the user input interface is based on the user input, and the VR device 200 responds to the user input through the controller 250 .

[0113] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 275, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0114] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0115] The memory 260 includes storing various software modules for driving the VR device 200. For example, the various software modules stored in the first memory include at least one of a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

[0116] The basic module is a low-level software module used for signal communication between various hardware in the VR device 200 and sending processing and control signals to the upper-level module. The detection module is a management module used to collect various information from various sensors or user input interfaces, and perform digital-to-analog conversion and analysis management.

[0117] For example, the speech recognition module includes a speech analysis module and a speech instruction database module. The display control module is a module for controlling the display to display image content, and can be used to play multimedia image content and UI interface information. The communication module is a module for controlling and communicating data with external devices. The browser module is a module for executing data communication between browsing servers. The service module is a module for providing various services and various applications. At the same time, the memory 260 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effects of focus objects.

[0118] Figure 3 Schematic diagram of the configuration of a handle according to some embodiments is shown. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0119] The handle 100 is configured to control the VR device 200, and can receive the user's input operation instructions, and convert the operation instructions into instructions that the VR device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the VR device 200. For example, the user operates the channel increase and decrease keys on the handle 100, and the VR device 200 responds to the channel increase and decrease operations.

[0120] In some embodiments, the handle 100 may be a smart device, for example, the handle 100 may be installed with various applications for controlling the VR device 200 according to user needs.

[0121] In some embodiments, Figure 1 As shown, after installing the application for controlling the VR device 200, the mobile terminal or other smart electronic device can play a similar function as the handle 100. For example, the user can install the application to realize the functions of the physical buttons of the handle 100 through various function keys or virtual buttons of the graphical user interface provided on the mobile terminal 300 or other smart electronic device.

[0122] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller is used to control the operation and operation of the handle 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0123] The communication interface 130, under the control of the controller 110, realizes the communication of control signals and data signals with the VR device 200. For example, the received user input signal is sent to the VR device 200. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0124] The user input / output interface 140 includes at least one of other input interfaces such as a microphone 141, a touch panel 142, a sensor 143, and a button 144. For example, the user can implement the user command input function through voice, touch, gesture, pressing, and other actions. The input interface converts the received analog signal into a digital signal, and the digital signal into a corresponding command signal, and sends it to the VR device 200.

[0125] The output interface includes an interface for sending the received user command to the VR device 200. In some embodiments, it can be an infrared interface or a radio frequency interface. For example, in the case of an infrared signal interface, the user input command needs to be converted into an infrared control signal according to the infrared control protocol, and sent to the VR device 200 through the infrared sending module. For another example, in the case of a radio frequency signal interface, the user input command needs to be converted into a digital signal, and then modulated according to the radio frequency control signal modulation protocol, and then sent to the VR device 200 by the radio frequency sending terminal.

[0126] In some embodiments, the handle 100 includes at least one of a communication interface 130 and an input / output interface 140. The handle 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, which can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the VR device 200.

[0127] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 200 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0128] The power supply 180 is used to provide operating power support for each component of the handle 100 under the control of the controller, and can be a battery and related control circuits.

[0129] In some embodiments, the handle 100 also includes an indicator light, which is used to assist the VR device 100 in positioning the handle 200 .

[0130] In some embodiments, the system may include a kernel, a command parser (shell), a file system, and an application. The kernel, shell, and file system together form the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. After startup, the application is compiled into machine code to form a process.

[0131] See also Figure 4In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0132] In some embodiments, at least one application is running in the application layer, and these applications may be window programs, system settings programs, clock programs, camera applications, etc. provided by the operating system; or they may be applications developed by third-party developers, such as hi-view programs, karaoke programs, magic mirror programs, etc. In specific implementation, the application packages in the application layer are not limited to the above examples, and may actually include other application packages, which are not limited in the embodiments of the present application.

[0133] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Applications can access system resources and obtain system services during execution through the API interface.

[0134] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0135] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit of the application (including switching the user interface currently displayed in the display window to the system desktop), opening, and backing (including switching the user interface currently displayed in the display window to the parent user interface of the currently displayed user interface), etc.

[0136] In some embodiments, the window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, capturing the screen, controlling display window changes (for example, shrinking the display window, shaking the display, distorting the display, etc.), etc.

[0137] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0138] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.), etc.

[0139] In some embodiments, the core layer further includes a power driver module for performing power management.

[0140] In some embodiments, Figure 4 The software programs and / or modules corresponding to the software architecture in Figure 2 or Figure 3 in the first memory or the second memory shown.

[0141] In some embodiments, taking the magic mirror application (photographing application) as an example, when the remote control receiving device receives the input operation of the remote control, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the input operation into a raw input event (including the value of the input operation, the timestamp of the input operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, identifies the control corresponding to the input event according to the current position of the focus, and the input operation is a confirmation operation, and the control corresponding to the confirmation operation is the control of the magic mirror application icon. The magic mirror application calls the interface of the application framework layer to start the magic mirror application, and then starts the camera driver by calling the kernel layer to realize the capture of static images or videos through the camera.

[0142] In some embodiments, for a display device with a touch function, taking a split-screen operation as an example, the display device receives an input operation (such as a split-screen operation) performed by a user on the display screen, and the kernel layer can generate a corresponding input event according to the input operation, and report the event to the application framework layer. The activity manager of the application framework layer sets the window mode (such as a multi-window mode) and the window position and size corresponding to the input operation. The window management of the application framework layer draws the window according to the settings of the activity manager, and then sends the drawn window data to the display driver of the kernel layer, and the display driver displays the corresponding application interface in different display areas of the display screen.

[0143] In some embodiments, Figure 5 As shown in , the application layer includes at least one application that can display a corresponding icon control in a VR device, such as: a live TV application icon control, a video on demand application icon control, a media center application icon control, an application center icon control, a game application icon control, etc.

[0144] In some embodiments, the live TV application can provide live TV through different signal sources. For example, the live TV application can use input from cable TV, wireless broadcast, satellite service, or other types of live TV services to provide TV signals. And, the live TV application can display video of the live TV signal on the VR device 200.

[0145] In some embodiments, the video on demand application can provide videos from different storage sources. Unlike the live TV application, the video on demand provides video display from certain storage sources. For example, the video on demand can come from the server side of the cloud storage, from the local hard disk storage containing the stored video programs.

[0146] In some embodiments, the media center application can provide various multimedia content playback applications. For example, the media center can be a service provided by the user to access various images or audio through the media center application, which is different from live TV or video on demand.

[0147] In some embodiments, the application center can provide storage for various applications. The application can be a game, an application, or some other application related to a computer system or other device but can be run on a smart TV. The application center can obtain these applications from different sources, store them in a local storage, and then run them on the VR device 200.

[0148] At present, virtual reality (VR) devices are widely used in education and training, fire drills, virtual driving, real estate and other fields, which can bring users an immersive visual feast. When using a virtual reality device, the movement trajectory of the handle connected to the VR device can usually be used to determine the user's instructions. Therefore, during the use of the virtual reality device, the handle needs to be positioned. In the prior art, during the positioning of the handle, the indicator light on the handle will flash. At the same time, the camera on the VR device will capture the image of the handle, and extract and encode the light spot formed in the image due to the flashing of the indicator light, and determine the light spot corresponding to the indicator light according to the coding information. Finally, the VR device completes the positioning of the handle according to the position of the light spot corresponding to the indicator light in the image. However, in a real environment, there may be other light sources around the handle, thereby forming an interfering light spot on the image of the handle, affecting the accuracy of the handle positioning.

[0149] To solve the above problems, the embodiments of the present application provide a virtual reality device and a handle positioning method, which removes interference spots in the image of the handle through the preset coding information of the indicator light on the handle, thereby improving the accuracy of handle positioning.

[0150] It can be understood that the above handle positioning method can be implemented by the virtual reality device provided in the embodiment of the present application. The technical solution of the embodiment of the present application is described in detail below by taking a virtual reality device integrated or installed with relevant execution code as an example. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0151] Figure 6 The flowchart of a handle positioning according to some embodiments is exemplarily shown. This embodiment involves a specific process of how to position the handle. Among them, the main execution diagram of this embodiment is a virtual reality device, and the virtual reality device is connected to the handle. Figure 6 As shown, the method includes:

[0152] S301, collecting multiple frames of images of a handle, where the handle is provided with at least one indicator light.

[0153] In the application embodiment, when the handle is connected to the VR device, if the VR device receives an instruction from the handle to position, multiple frames of images of the handle can be captured.

[0154] The embodiments of the present application do not limit how to capture multiple frames of images of the handle. In some embodiments, the VR device may carry a camera, and multiple frames of images of the handle may be captured by calling the camera carried on the VR device. In other embodiments, the VR device may also be externally connected to other camera devices. The VR device may call the external camera device to capture multiple frames of images of the handle by sending instruction information to the external camera device. Among them, the embodiments of the present application do not limit the type of camera. For example, it may be a monocular camera.

[0155] In the present application, at least one indicator light may be provided on the handle. When the VR device collects multiple frames of images of the handle, the indicator light on the handle may flash, so that the collected multiple frames of images contain light spots corresponding to the indicator light, thereby assisting the VR device in locating the handle. The present application embodiment does not limit the type of indicator light, and for example, it may be a light-emitting diode (LED).

[0156] It should be noted that the embodiment of the present application does not limit the number of indicator lights and the arrangement of the indicator lights on the handle, and can be specifically set according to actual conditions. Figure 7a-7c The following is a schematic diagram showing the arrangement of indicator lights according to some embodiments. Figure 7a-7c As shown, for handles of different shapes, the indicator lights can be arranged in sequence according to the edge of the handle. It should be understood that the colors of the indicator lights in the embodiment of the present application can be the same or different, and can be specifically set according to actual conditions.

[0157] It should be understood that in order to make the indicator light flash according to a preset flashing cycle, in some embodiments, before collecting multiple frames of images of the handle, the VR device can periodically send synchronization information to the handle, and the synchronization information includes the flashing cycle of at least one indicator light. After receiving the synchronization information, the handle can control the indicator light to flash according to the flashing cycle. At the same time, the VR device can synchronously record images so that the image of the handle captured by the camera exposure is exactly the image when the indicator light of the handle is on.

[0158] S302: extract at least one light spot from multiple frames of images of the handle.

[0159] In this step, after the VR device captures multiple frames of images of the handle, the controller in the VR device can extract at least one light spot from the multiple frames of images of the handle.

[0160] The embodiments of the present application do not limit how to extract light spots. In some embodiments, the controller in the VR device can sequentially input multiple frames of images of the handle into the image recognition algorithm model to obtain at least one light spot extracted from each frame of the image input to the image recognition algorithm model. Exemplarily, the image recognition algorithm model can be the opecv blob light spot extraction algorithm model. The opecv blob light spot extraction algorithm model can analyze the connected domains of the same pixels in the image and extract the light spots in the image after binarizing the image.

[0161] S303. Encode at least one light spot to form first encoded information.

[0162] In this step, when the controller in the VR device extracts at least one light spot from multiple frames of images of the handle, it can encode the at least one light spot to form first encoded information.

[0163] In some embodiments, the controller in the VR device can first determine the brightness and darkness changes of at least one light spot in multiple frames of images according to the diameter changes of the at least one light spot in the multiple frames of images. Subsequently, the controller in the VR device encodes the at least one light spot according to the brightness and darkness changes in the multiple frames of images to form first encoded information.

[0164] Exemplarily, Figure 8a-8b Exemplarily shows a schematic diagram of a bright spot according to some embodiments, as shown in FIGS. 8a - 8b. Figure 8a The diameter of the shown bright spot can be set to R0. Figure 8b The diameter of the shown bright spot can be set to R1. Figure 8a For Figure 8b the previous frame of the image.

[0165] If R0 > m * R1, it is determined that the diameter of the light spot changes from large to small, and its corresponding indicator light changes from bright to dark. Accordingly, the encoding of the light spot in Figure 8a can be set to 1, and the encoding in Figure 8b can be set to 0. Wherein, m is a constant and can be specifically set according to the actual situation. For example, m is 1.3, 1.5, etc.

[0166] If R0 < n * R1, it is determined that the diameter of the light spot changes from small to large, and its corresponding indicator light changes from dark to bright. Accordingly, the encoding of the light spot in Figure 8a can be set to 0, and the encoding in Figure 8b can be set to 1. Wherein, n is a constant and can be specifically set according to the actual situation. For example, m is 1.3, 1.5, etc.

[0167] In the embodiment of the present application, after the encoding of each bright spot in each frame of the image is completed, the encoding information of the same bright spot can be combined in time order to form the first encoding information corresponding to the bright spot. It should be understood that the embodiment of the present application does not limit the number of frames of the image contained in the first encoding information, and illustratively, it can be five frames.

[0168] S304: Remove interference light spots from at least one light spot according to the first coding information and the second coding information of at least one indicator light, and determine a light spot corresponding to the at least one indicator light.

[0169] In this step, after the VR device determines the first coding information, the first coding information can be compared with the second coding information of at least one indicator light pre-set in the VR device, so as to remove the interfering light spot in at least one light spot and determine the light spot corresponding to the at least one indicator light.

[0170] Exemplarily, Table 1 is a second coding information table of the indicator light.

[0171] Table 1

[0172]

[0173] In the present application, if there is second coding information identical to the first coding information, the light spot corresponding to the first coding information is determined to be a light spot corresponding to at least one indicator light. Correspondingly, if there is no second coding information identical to the first coding information, the light spot corresponding to the first coding information is determined to be an interference light spot, and then the interference light spot is removed.

[0174] S305: Determine the position of the handle according to the position of the light spot corresponding to at least one indicator light in the multiple frames of images.

[0175] In this step, after the controller in the VR device removes the interfering light spot in at least one light spot and determines the light spot corresponding to at least one indicator light, the position of the handle can be determined according to the position of the light spot corresponding to the at least one indicator light in multiple frame images.

[0176] In some embodiments, the controller in the VR device can determine the spatial layout of the indicator lights on the handle according to the position of the light spot corresponding to at least one indicator light in multiple frames of images. Then, the position of the handle in the three-dimensional control is determined through a perspective-n-point (PnP) algorithm, thereby achieving the positioning and tracking of the handle.

[0177] It should be understood that the positioning and tracking of the handle in the present application can be used not only for 3DOF (coordinates) but also for 6DOF, and the embodiments of the present application are not limited to this.

[0178] In other possible implementations, after completing the positioning and tracking of the handle, the position of the handle may be further smoothed and predicted, thereby improving the timeliness and smoothness of the positioning and tracking of the handle.

[0179] The handle positioning method provided in the embodiment of the present application is provided with at least one indicator light by collecting multiple frames of images of the handle. Secondly, at least one light spot is extracted from the multiple frames of images of the handle, and the at least one light spot is encoded to form a first coding information. Thirdly, according to the first coding information and the second coding information of at least one indicator light, the interfering light spot in the at least one light spot is removed, and the light spot corresponding to the at least one indicator light is determined. Finally, according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images, the position of the handle is determined. Compared with the prior art, the embodiment of the present application can remove the interfering light spot by the coding information preset by the indicator light when positioning the handle, thereby improving the accuracy of the handle positioning.

[0180] Based on the above embodiment, the following describes how to remove interference spots in the handle image of the virtual reality device. Fig. 9 A schematic diagram of another process of positioning a handle according to some embodiments is exemplarily shown. Fig. 9 As shown, the method includes:

[0181] S401, collecting multiple frames of images of a handle, where the handle is provided with at least one indicator light.

[0182] S402: extract at least one light spot from multiple frames of images of the handle.

[0183] The technical terms, technical effects, technical features, and optional implementation methods of S401-S402 can be found in Figure 6 The S301 - S302 shown are understood, and the repeated contents will not be repeated here.

[0184] S403 . Determine brightness changes of at least one light spot in the multiple frames of images according to diameter changes of at least one light spot in the multiple frames of images.

[0185] S404: Encode at least one light spot according to the brightness changes in the multiple frames of images to form first encoding information.

[0186] In some embodiments, at least one light spot is encoded according to the brightness changes in multiple frames of images and the correspondence between the brightness and darkness of the light spot and the bitmap information to form first encoding information.

[0187] For example, continue to refer to Figure 8a-8b The correspondence between the brightness of the light spot and the bitmap information can be, for example, for the same light spot in two adjacent frames of images, the brighter one corresponds to 1 and the darker one corresponds to 0.

[0188] S405: Determine whether there is second coding information that is the same as the first coding information.

[0189] If yes, execute step S406, if no, execute step S407.

[0190] S406: Determine that the light spot corresponding to the first coding information is a light spot corresponding to at least one indicator light.

[0191] S407: Determine that the light spot corresponding to the first coding information is an interference light spot.

[0192] S408, removing the interfering light spots.

[0193] S409: Determine the position of the handle according to the position of the light spot corresponding to at least one indicator light in the multiple frames of images.

[0194] The handle positioning method provided in the embodiment of the present application is provided with at least one indicator light by collecting multiple frames of images of the handle. Secondly, at least one light spot is extracted from the multiple frames of images of the handle, and the at least one light spot is encoded to form a first coding information. Thirdly, according to the first coding information and the second coding information of at least one indicator light, the interfering light spot in the at least one light spot is removed, and the light spot corresponding to the at least one indicator light is determined. Finally, according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images, the position of the handle is determined. Compared with the prior art, the embodiment of the present application can remove the interfering light spot by the coding information preset by the indicator light when positioning the handle, thereby improving the accuracy of the handle positioning.

[0195] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0196] Fig.10 The structure diagram of a virtual reality device according to some embodiments is exemplarily shown. The virtual reality device can be implemented by software, hardware, or a combination of both to execute the handle positioning method in the above embodiment. Fig.10 As shown, the virtual reality device 500 includes: a display 501, a camera 502 and a controller 503.

[0197] The camera is configured to collect multiple frames of images of the handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light;

[0198] The controller connected to the camera is configured as follows:

[0199] Extracting at least one light spot from multiple frames of images of the handle;

[0200] Encoding at least one light spot to form first coded information;

[0201] According to the first coding information and the second coding information of the at least one indicator light, removing the interfering light spot in the at least one light spot, and determining the light spot corresponding to the at least one indicator light;

[0202] The position of the handle is determined according to the position of a light spot corresponding to at least one indicator light in the multiple frames of images.

[0203] In some embodiments of the present application, the controller is specifically configured as follows:

[0204] Determining brightness changes of at least one light spot in the multiple frames of images according to a diameter change of at least one light spot in the multiple frames of images;

[0205] According to the brightness and darkness changes in the multiple frames of images, at least one light spot is encoded to form first encoding information.

[0206] In some embodiments of the present application, the controller is specifically configured as follows:

[0207] According to the brightness changes in the multiple frames of images, and the corresponding relationship between the brightness and darkness of the light spots and the bitmap information, at least one light spot is encoded to form first encoding information.

[0208] In some embodiments of the present application, the controller is specifically configured as follows:

[0209] If there is second coding information that is the same as the first coding information, determining that the light spot corresponding to the first coding information is the light spot corresponding to at least one indicator light;

[0210] If there is no second coding information identical to the first coding information, determining that the light spot corresponding to the first coding information is an interference light spot;

[0211] Remove interfering light spots.

[0212] In some embodiments of the present application, the controller is further configured to:

[0213] Send synchronization information to the handle, where the synchronization information includes a flashing cycle of at least one indicator light.

[0214] The virtual reality device provided in the embodiment of the present application can execute the actions of the handle positioning method in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0215] The embodiment of the present application also provides a program, which, when executed by a processor, is used to execute the handle positioning method provided in the above method embodiment.

[0216] An embodiment of the present application also provides a program product, such as a computer-readable storage medium, in which instructions are stored. When the program product is run on a computer, the computer executes the handle positioning method provided by the above method embodiment.

[0217] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual reality device, characterized in that: include, monitor; A camera is configured to collect multiple frames of images of a handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light; A controller connected to the camera, the controller being configured to: Extracting at least one light spot from the multiple frames of images of the handle; Encoding the at least one light spot to form first coded information; According to the first coding information and second coding information of the at least one indicator light preset in the virtual reality device, removing interference light spots in the at least one light spot, and determining the light spot corresponding to the at least one indicator light; Determining the position of the handle according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images; The controller is specifically configured as follows: Determining brightness changes of the at least one light spot in the multiple frames of images according to changes in diameter of the at least one light spot in the multiple frames of images; According to the brightness changes in the multiple frames of images, and the corresponding relationship between the brightness and darkness of the light spot and the bitmap information, the at least one light spot is encoded to form first encoding information; The controller is specifically configured as follows: If there is second coding information that is the same as the first coding information, determining that the light spot corresponding to the first coding information is the light spot corresponding to the at least one indicator light; If there is no second coding information identical to the first coding information, determining that the light spot corresponding to the first coding information is an interference light spot; The interfering light spot is removed.

2. The virtual reality device according to claim 1, characterized in that: The controller is also configured to: Sending synchronization information to the handle, wherein the synchronization information includes a flashing cycle of the at least one indicator light.

3. A handle positioning method, characterized in that: Applied to a virtual reality device, the virtual reality device is connected to the handle, and the method includes: Collecting multiple frames of images of the handle, wherein the handle is provided with at least one indicator light; Extracting at least one light spot from the multiple frames of images of the handle; Encoding the at least one light spot to form first coded information; According to the first coding information and second coding information of the at least one indicator light preset in the virtual reality device, removing interference light spots in the at least one light spot, and determining the light spot corresponding to the at least one indicator light; Determining the position of the handle according to the position of the light spot corresponding to the at least one indicator light in the multiple frames of images; The step of encoding the at least one light spot to form first coded information includes: Determining brightness changes of the at least one light spot in the multiple frames of images according to changes in diameter of the at least one light spot in the multiple frames of images; According to the brightness changes in the multiple frames of images, and the corresponding relationship between the brightness and darkness of the light spot and the bitmap information, the at least one light spot is encoded to form first encoding information; The removing the interfering light spot in the at least one light spot according to the first coding information and the second coding information of the at least one indicator light preset in the virtual reality device, and determining the light spot corresponding to the at least one indicator light, comprises: If there is second coding information that is the same as the first coding information, determining that the light spot corresponding to the first coding information is the light spot corresponding to the at least one indicator light; If there is no second coding information identical to the first coding information, determining that the light spot corresponding to the first coding information is an interference light spot; The interfering light spot is removed.

4. The method according to claim 3, characterized in that Before acquiring the multiple frames of images of the handle, the method further includes: Sending synchronization information to the handle, wherein the synchronization information includes a flashing cycle of the at least one indicator light.

Images