Virtual reality apparatus, method, and storage medium

Abstract

The application provides a virtual reality device, a method and a storage medium, and belongs to the technical field of virtual reality. The virtual reality device provided by the application is provided with a plurality of cameras and a plurality of processing chips connected through a bus, and each processing chip is connected with part of the plurality of cameras, so that the plurality of processing chips share the virtual reality display task, the performance requirement of a single processing chip is low, and the cost can be effectively reduced.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a virtual reality device, method, and storage medium. Background Technology

[0002] With the development of VR (Virtual Reality) technology, people have higher and higher requirements for VR devices. Correspondingly, VR devices also have higher and higher requirements for computing resources when performing high-load tasks such as image processing.

[0003] Current VR devices use high-performance processors to process the images captured by the VR device. However, due to the high price of high-performance processors, the cost of the aforementioned VR devices is high. There is an urgent need for a virtual reality device that can reduce costs while meeting the computing resource requirements. Summary of the Invention

[0004] This application provides a virtual reality device, method, and storage medium to reduce costs while meeting computing resource requirements. The technical solution is as follows:

[0005] In a first aspect, a virtual reality device is provided, the virtual reality device comprising:

[0006] Wearable pendant and display device, the wearable pendant and display device are connected;

[0007] The wearable pendant includes multiple cameras and multiple processing chips mounted on it. The multiple processing chips are connected to each other via a bus. Each processing chip is connected to a portion of the multiple cameras. The multiple processing chips are used to perform a portion of the virtual reality display task based on the images captured by the multiple cameras, so as to obtain virtual reality images and transmit the virtual reality images to the display device.

[0008] Display devices are used to display virtual reality images.

[0009] The virtual reality device provided in this application includes multiple cameras and multiple processing chips connected via a bus. Each processing chip is connected to some of the cameras, so that multiple processing chips share and coordinate the virtual reality display task. This reduces the performance requirements of a single processing chip and effectively lowers the cost.

[0010] Optionally, the plurality of processing chips includes a first processing chip, which is used to perform spatial modeling tasks in virtual reality display tasks;

[0011] The first processing chip is connected to a modeling camera group among multiple cameras. The first processing chip is also connected to multiple interface switches. Each interface switch is connected to at least one camera in the modeling camera group. The cameras in the modeling camera group are used to capture images of the environment within a corresponding angle range. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

[0012] Optionally, multiple cameras in the modeling camera group take pictures of the environment within the corresponding angle range in a preset shooting order.

[0013] Optionally, the wearable pendant has a ring-shaped structure, and the cameras in the modeling camera group are distributed on the outer ring of the wearable pendant according to preset positions, so that the shooting range of the cameras in the modeling camera group covers the angle range corresponding to the wearable pendant.

[0014] Optionally, the first processing chip is also used to control the camera corresponding to the abnormal situation to shoot at a preset frame rate through a corresponding interface switch when an abnormal situation is detected based on the images captured by the cameras in the modeling camera group.

[0015] Optionally, the first processing chip is also used to control the corresponding interface switch to switch the camera that is taking pictures to the camera corresponding to the abnormal situation when a change in the camera corresponding to the abnormal situation is detected based on the images captured by the cameras in the modeling camera group.

[0016] Optionally, the plurality of processing chips includes a second processing chip for performing spatial audio tasks in virtual reality display tasks;

[0017] The second processing chip is connected to a spatial audio camera group among multiple cameras. The second processing chip is also connected to multiple interface switches. Each interface switch is connected to at least one camera in the spatial audio camera group. The cameras in the spatial audio camera group are used to capture images of the human ear. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

[0018] Optionally, the plurality of processing chips includes a third processing chip, which is used to perform gesture interaction tasks in virtual reality display tasks;

[0019] The third processing chip is connected to a gesture camera group among multiple cameras. The third processing chip is also connected to multiple interface switches. Each interface switch is connected to at least one camera in the gesture camera group. The cameras in the gesture camera group are used to capture images of a preset gesture area. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

[0020] Optionally, the plurality of processing chips includes a fourth processing chip, which is used to perform perspective tasks in virtual reality display tasks;

[0021] The fourth processing chip is connected to a perspective camera group among multiple cameras, which are used to capture images of the environment within the field of view of the human eye.

[0022] Optionally, the plurality of processing chips includes a fifth processing chip, which is used to perform eye-tracking tasks in virtual reality display tasks;

[0023] The fifth processing chip is connected to a tracking camera group among multiple cameras, which are used to capture images of the human eye.

[0024] Optionally, the plurality of processing chips includes a sixth processing chip, which is used to receive the processing result obtained by any of the plurality of processing chips and to perform the rendering task in the virtual reality display task on the processing result.

[0025] Optionally, any one of the multiple processing chips is also used to send images captured by the cameras in a camera group to the processing chip corresponding to the task of the camera group when connected to a camera in a camera group that does not match the task being processed by the processing chip.

[0026] Optionally, any of the multiple processing chips may also be used to send the processing result to the processing chip corresponding to the next task after completing the task corresponding to the processing chip.

[0027] Optionally, the multiple cameras include at least one main camera and at least one slave camera corresponding to each main camera;

[0028] The slave camera is used to respond to the main camera corresponding to the slave camera starting to shoot, and shoots synchronously with the main camera.

[0029] Optionally, multiple cameras shooting simultaneously may not be connected to the same interface switch.

[0030] Optionally, the plurality of processing chips includes at least one main processing chip and a corresponding slave processing chip for each main processing chip;

[0031] The main processing chip is used to control the operation starting from the processing chip.

[0032] Optionally, the processing chip is also used to identify images captured by the camera based on the camera's identifier.

[0033] Secondly, a virtual reality display method is provided. The virtual reality device includes a wearable pendant and a display device, the wearable pendant and the display device are connected, the wearable pendant includes multiple cameras and multiple processing chips disposed on the outer ring of the wearable pendant, the multiple processing chips are connected to each other via a bus, and each processing chip is connected to a portion of the multiple cameras. The method includes:

[0034] Multiple processing chips execute a portion of the virtual reality display task based on images captured by multiple cameras to obtain virtual reality images and transmit them to the display device;

[0035] Display devices display virtual reality images.

[0036] Thirdly, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to perform the operations performed by the virtual reality display method provided in the second aspect or various alternative implementations of the second aspect.

[0037] Fourthly, a computer program product is provided, comprising a computer program that is executed by a processor to perform the operations performed by the virtual reality display method provided in the second aspect or various alternative implementations of the second aspect.

[0038] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the structure of a virtual reality device provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of another virtual reality device provided in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the camera distribution in a modeling camera group provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of a master-slave exposure method provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram illustrating an exposure method provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of an interface switch provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the camera distribution in a gesture camera group and a spatial audio camera group provided in an embodiment of this application;

[0047] Figure 8 This is a connection diagram of a processing chip provided in an embodiment of this application;

[0048] Figure 9 This is a framework diagram of an on-chip application provided in an embodiment of this application;

[0049] Figure 10 This is a data interaction diagram of a virtual reality display method provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0051] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0052] In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0053] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the images and instructions involved in this application were obtained with full authorization.

[0054] Figure 1 This is a schematic diagram of the structure of a virtual reality device provided in an embodiment of this application, as shown below. Figure 1 As shown, the virtual reality device includes a wearable pendant 101 and a display device 102, which are connected together. The wearable pendant 101 can be a neck pendant, such as a U-shaped pillow, and the display device 102 can be a head-mounted display, such as VR glasses. This application embodiment does not limit the specific implementation of the device.

[0055] The wearable pendant 101 includes multiple cameras and multiple processing chips mounted on it. These processing chips are connected via a bus, and each processing chip is connected to a portion of the cameras. Based on the images captured by the multiple cameras, each processing chip performs a portion of the virtual reality display task to obtain a virtual reality image, which is then transmitted to the display device 102. The display device 102 then displays the virtual reality image.

[0056] The multiple cameras can be implemented as digital cameras or infrared cameras, etc., and the processing chip can be implemented as a SoC (System on Chip). The SoC integrates a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Network Processing Unit), etc. This application embodiment does not limit the specific implementation. The CPU, GPU, and NPU can be used to perform the same or different tasks, for example... Figure 2 In this system, the NPU is used to perform gesture interaction tasks, eye tracking tasks, and spatial modeling tasks (also known as neck perspective tasks); the CPU is used to perform perspective tasks, spatial audio tasks, and spatial modeling tasks; and the GPU is used to perform perspective tasks. The display device includes at least one display screen; for example, the display device includes two display screens, with the left screen corresponding to the left eye and the right screen corresponding to the right eye. The resolution of the display screen can be 4K, etc., and this embodiment does not limit this. The virtual reality image is obtained by the processing chip processing the image captured by the camera according to the algorithm corresponding to the task.

[0057] In some embodiments, virtual reality display tasks include spatial modeling tasks, spatial audio tasks, gesture interaction tasks, perspective tasks, and eye-tracking tasks, etc. Each of these tasks corresponds to a camera group, and each camera group includes at least one camera used to capture images required for the task corresponding to the camera group. For example, a spatial modeling task corresponds to a modeling camera group, a spatial audio task corresponds to a spatial audio camera group, a gesture interaction task corresponds to a gesture camera group, a perspective task corresponds to a perspective camera group, and an eye-tracking task corresponds to a tracking camera group. Taking the spatial modeling task as an example, the cameras in the modeling camera group are used to capture images of the environment surrounding the virtual reality device, so that during the processing of the spatial modeling task, the processing chip can perform spatial modeling based on the images captured by the cameras in the modeling camera group. The following... Figure 2The content shown provides a detailed description of the multiple tasks included in the aforementioned virtual reality display task, which will not be elaborated upon further in this embodiment. A camera may belong to one camera group, or one camera may belong to multiple camera groups; this embodiment does not limit this to any particular category.

[0058] In some embodiments, the virtual reality display task further includes a rendering task, which is used to render the image obtained by performing the above task, or to render the image or audio based on the result obtained by performing the above task. For example, performing the above spatial modeling task refers to using a spatial modeling algorithm to process the image captured by the camera to obtain a fused image, while performing the rendering task refers to rendering the fused image obtained by performing the spatial modeling task to obtain a virtual reality image.

[0059] In some embodiments, each of the plurality of cameras is connected to a processing chip. In the case of a camera group including a plurality of cameras, the plurality of cameras in the camera group are connected to the same processing chip, or the plurality of cameras in the camera group are connected to different processing chips. This application embodiment does not limit this.

[0060] In some embodiments, each of the plurality of processing chips is used to perform a portion of the tasks in the virtual reality display task. A single processing chip performs one task in the virtual reality display task, or a single processing chip performs multiple tasks in the virtual reality display task. This application embodiment does not limit this. Correspondingly, the cameras in the camera group are connected to corresponding processing chips, or connected to non-corresponding processing chips. This application embodiment does not limit this. A processing chip corresponding to a camera means that the processing chip is used to perform the task corresponding to that camera, and a processing chip not corresponding to a camera means that the processing chip is not used to perform the task corresponding to that camera. When a camera is connected to a non-corresponding processing chip, after the camera sends the captured image to the processing chip, the processing chip sends the image to the processing chip performing the corresponding task, so that the processing chip performing the corresponding task processes the image. For example, a camera in the modeling camera group is connected to a processing chip performing a rendering task. After receiving an image captured by the camera, the processing chip sends the image to a processing chip performing a spatial modeling task, so that the processing chip performing the spatial modeling task processes the image.

[0061] In some embodiments, during the execution of a virtual reality display task, some tasks within the virtual reality display task are executed sequentially to complete a full virtual reality display process. Correspondingly, when a certain task is executed by different processing chips, the processing chips corresponding to these tasks sequentially execute the current corresponding task based on the execution result of the previous task, according to the execution order of the tasks. Correspondingly, the processing chip corresponding to any of the aforementioned tasks receives the execution result of the processing chip corresponding to the previous task, executes the task corresponding to that processing chip based on that result, and sends the execution result of the current processing chip to the processing chip corresponding to the next task. For example, if the spatial modeling task and the rendering task in the aforementioned virtual reality display task are executed sequentially to complete a full virtual reality display process, and the processing chip executing the spatial modeling task and the processing chip executing the rendering task are different processing chips, then the processing chip executing the spatial modeling task receives an image sent by a camera, performs a spatial modeling task on that image to obtain a fused image, and sends the fused image to the processing chip executing the rendering task. The processing chip executing the rendering task renders the fused image to obtain a virtual reality image.

[0062] The virtual reality device provided in this application embodiment is equipped with multiple cameras and multiple processing chips connected by a bus. Each processing chip is connected to some of the cameras, so that multiple processing chips share and coordinate the virtual reality display task. The performance requirements of a single processing chip are low, which can effectively reduce costs.

[0063] The above content provides a brief description of the virtual reality device provided in the embodiments of this application. The following description, in conjunction with... Figure 2 The detailed structure of the aforementioned virtual reality device will be described by way of example. Figure 2 This is a schematic diagram of the structure of a virtual reality device provided in an embodiment of this application, as shown below. Figure 2As shown, the virtual reality device includes two processing chips, AP1 and AP2, and 20 cameras, designated as Cameras 1 to 20. The cameras are assigned according to their functions: Cameras 1 to 12 are in the modeling camera group; Cameras 13 and 14 are in the gesture camera group; Cameras 15 and 16 are in the spatial audio camera group; Cameras 17 and 18 are in the perspective camera group; and Cameras 19 and 20 are in the tracking camera group. AP1 and AP2 each have six MIPI (Mobile Industry Processor Interface) interfaces (camera interfaces) for connecting to the cameras. AP1 is the processing chip for performing rendering tasks, while AP2 is the processing chip for performing spatial modeling, gesture interaction, spatial audio, and eye-tracking tasks.

[0064] The workflow of the virtual reality device will be described below through the tasks corresponding to the aforementioned camera groups.

[0065] (1) Spatial modeling task

[0066] The following section, in conjunction with the spatial modeling task, explains the distribution of cameras in the modeling camera group and the connection between the cameras and the processing chip. The cameras in the modeling camera group are used to capture images of the environment within the corresponding angular range.

[0067] In some embodiments, such as Figure 3 As shown, the wearable accessory in the virtual reality device has a ring-shaped structure. Figure 2 The 12 cameras in the modeling camera group shown are distributed at preset positions on the outer ring of the wearable pendant. Each camera is used to capture the environment within a preset angle range around the wearable pendant, so that the shooting range of the 12 cameras can cover the angle range corresponding to the ring-shaped wearable pendant. This allows the cameras in the modeling camera group to capture as much of the environment around the wearable pendant as possible, thereby enabling a wider range of detection of the user's surrounding environment, timely detection of abnormal situations and alerting the user, and improving the safety of the user when using the virtual reality device provided in this application embodiment. The aforementioned distribution of the 12 cameras at preset positions on the outer ring of the wearable pendant means either that the 12 cameras are evenly distributed on the outer ring of the wearable pendant, or that the 12 cameras are distributed on the outer ring of the wearable pendant according to the viewing angle of each camera, etc., so that the shooting range of the 12 cameras covers the angle range corresponding to the ring-shaped wearable pendant. This application embodiment does not limit this specific distribution.

[0068] In some embodiments, the cameras in the modeling camera group are connected to a processing chip that performs spatial modeling tasks, or the cameras in the modeling camera group are connected to a processing chip that does not perform spatial modeling tasks. Figure 2 As shown, AP2 is the processing chip that performs spatial modeling tasks, and AP1 is the processing chip that performs rendering tasks. Cameras 1, 2, 4-9, and 11-12 are connected to AP2, while cameras 3 and 10 are connected to AP1. Of course, the above... Figure 2 The connection method between the camera and the processing chip shown is only an example. In some embodiments, all cameras in the above-described modeling camera group are connected to AP1, or all are connected to AP2, or some are connected to AP1 and some are connected to AP2. This application embodiment does not limit this.

[0069] In some embodiments, when a camera in a modeling camera group is connected to a processing chip that performs a spatial modeling task, the camera in the modeling camera group captures images of the environment within a corresponding angular range, obtains an image, and sends the image to the processing chip connected to the camera. Upon receiving the image, the processing chip performs a spatial modeling task based on the image. For example, as... Figure 2 As shown, camera 1 is the camera in the modeling camera group, and camera 1 is connected to AP2. After camera 1 captures an image of the environment within a corresponding angular range, it sends the image to AP2, which then performs a spatial modeling task based on the image. During the spatial modeling task, the processing chip uses a pre-set spatial modeling algorithm to process the images captured by the cameras in the modeling camera group to model the environment surrounding the virtual reality device and obtain a fused image.

[0070] In some embodiments, when a camera in a modeling camera group is connected to a processing chip that does not perform spatial modeling tasks, after the camera in the modeling camera group captures an image, it sends the captured image to the processing chip connected to that camera. The processing chip then forwards the image to the processing chip that performs the spatial modeling task, so that the processing chip performing the spatial modeling task can perform the spatial modeling task based on the image. For example, such as... Figure 2 As shown, camera 3 is the camera in the modeling camera group, and camera 3 is connected to AP1. After camera 3 takes a picture of the environment within the corresponding angle range, it sends the image to AP1. AP1 forwards the image to AP2, and AP2 performs the spatial modeling task based on the image.

[0071] In some embodiments, the virtual reality device sequentially performs spatial modeling and rendering tasks to obtain a virtual reality image, which is used to display the environment surrounding the virtual reality device. When the processing chip performing the spatial modeling task and the processing chip performing the rendering task are different processing chips, the processing chip performing the spatial modeling task completes the spatial modeling task based on the images sent by the camera and obtains the execution result, i.e., the fused image. The processing chip performing the spatial modeling task sends the execution result to the processing chip performing the rendering task, which then renders the execution result to obtain the virtual reality image. For example, as... Figure 2 As shown, AP2 performs a spatial modeling task based on images captured by at least one of cameras 1-12, and obtains a fused image. AP2 then sends the fused image to AP1, which renders the fused image to obtain a virtual reality image.

[0072] In some embodiments, the processing chip that performs the rendering task is connected to the display device. After rendering the fused image to obtain a virtual reality image, the processing chip that performs the rendering task sends the virtual reality image to the display device so that the display device can display the virtual reality image.

[0073] In some embodiments, multiple cameras in the modeling camera group are connected to a processing chip via pathways connected by interface switches. Correspondingly, the processing chip is connected to multiple interface switches, and each interface switch is connected to at least one camera in the modeling camera group. This processing chip may be a processing chip that performs spatial modeling tasks, or it may be a processing chip that does not perform spatial modeling tasks; this application embodiment does not limit this. For example, as... Figure 2 As shown, AP2 is connected to interface switches 1-6. Interface switch 1 is also connected to camera 1, interface switch 2 is also connected to camera 12, interface switch 3 is also connected to cameras 2 and 5, interface switch 4 is also connected to cameras 4 and 6, interface switch 5 is also connected to cameras 7 and 9, and interface switch 6 is also connected to cameras 8 and 11. AP1 is connected to interface switches 7 and 8. Interface switch 7 is also connected to camera 3, and interface switch 8 is also connected to camera 7.

[0074] The above process involves multiple processing chips, including a first processing chip, which performs spatial modeling tasks in virtual reality display. The first processing chip is connected to a modeling camera group among multiple cameras. It is also connected to multiple interface switches, each interface switch being connected to at least one camera in the modeling camera group. The interface switches control the cameras connected to them to capture images at different times. In this implementation, the first processing chip is AP2. Figure 2In the virtual reality device shown, each camera in the modeling camera group is connected to the processing chip through a path connected by an interface switch. In some embodiments, some cameras in the modeling camera group are directly connected to the interface chip, but this application does not limit this.

[0075] In some embodiments, an interface switch is connected to one or more cameras, and multiple cameras connected to the same interface switch belong to the same or different camera groups. For example, such as Figure 2 As shown, interface switch 1 is connected to camera 1 and camera 13, which belong to the modeling camera group and the gesture camera group, respectively.

[0076] The following describes the workflow of the virtual reality device during the execution of the spatial modeling task.

[0077] In some embodiments, during the execution of a spatial modeling task, multiple cameras in the modeling camera group take turns exposing their images. That is, the multiple cameras in the modeling camera group sequentially capture images of the environment within a corresponding angular range according to a preset order, and send the captured images to the processing chip that processes the spatial modeling task. The processing chip processes the images captured by the multiple cameras in the modeling camera group according to a spatial modeling algorithm to obtain a fused image. After obtaining the fused image, the processing chip sends the fused image to the processing chip that processes the rendering task; alternatively, if the processing chip detects an abnormal situation based on the fused image, it sends the fused image to the processing chip that processes the rendering task. The processing chip that processes the rendering task renders the fused image to obtain a virtual reality image, and sends the virtual reality image to a display device so that the display device can display the virtual reality image.

[0078] In some embodiments, when the processing chip detects an anomaly based on the fused image, it sends the fused image to the processing chip handling the rendering task. During this process, the processing chip compares the fused image with a previously obtained fused image. If a new object appears in the fused image, the processing chip detects an anomaly and sends the fused image to the processing chip handling the rendering task. The processing chip handling the rendering task renders the fused image to obtain a virtual reality image and sends the virtual reality image to the display device. This allows the display device to alert the user to the anomaly, enabling the user to respond promptly and improving the safety of using the virtual reality device. If no new object appears in the fused image, the processing chip does not detect an anomaly and does not send the fused image to the processing chip handling the rendering task.

[0079] The preset order can be a left-to-right or right-to-left sequence on the wearable pendant, or a sequence of shots taken sequentially to the left or right from any camera in the modeling camera group, or a random sequence. This application embodiment does not limit this. Abnormal situations can include the presence of a dangerous object approaching, or the user approaching a dangerous object, etc. This application embodiment does not limit this.

[0080] In some embodiments, the processing chip compares the fused image with the previously obtained fused image. If no new object appears in the fused image, the processing chip does not detect any abnormality and retains the fused image for comparison with the next fused image.

[0081] For example, such as Figure 3 As shown, cameras 1-12 sequentially capture images of the environment within their respective angular ranges from left to right. Cameras 1, 2, 4-9, 11, and 12 send their captured images to AP2 in the order they were captured via an interface switch connected to AP2. Cameras 3 and 10 send their captured images to AP1 via an interface switch connected to AP1, and AP1 forwards these images to AP2. After completing one round of polling exposure, AP2 obtains 12 images captured by cameras 1-12. AP2 processes these 12 images using a spatial modeling algorithm to obtain a fused image. AP2 compares this fused image with the fused image corresponding to the previous round of polling exposure. If a new object appears in the current fused image, AP2 detects an anomaly and sends the current fused image to AP1. AP1 renders the current fused image to obtain a virtual reality image and sends this virtual reality image to the display device for display. If no new objects appear in the current fused image, AP2 does not detect any abnormalities. AP2 retains the fused image and does not send it to AP1. The display device does not need to display the virtual reality image obtained through the fused image.

[0082] In some embodiments, when a processing chip performing a spatial modeling task detects an abnormal situation, the processing chip identifies the camera corresponding to the abnormal situation and controls the camera to take pictures at a preset frame rate to continuously observe the abnormal situation and ensure user safety.

[0083] In the process of identifying the camera corresponding to an anomaly, the processing chip performs identification based on the fused image or based on images captured by individual cameras. When identifying based on the fused image, the processing chip identifies the coordinates corresponding to the anomaly from the fused image. Based on these coordinates, it obtains the position of the anomaly relative to the wearable pendant. Then, based on the angular range of this position, it identifies the camera corresponding to the anomaly. When identifying based on images captured by individual cameras, for each image, the processing chip compares it with the previous image captured by the corresponding camera. If a new object appears in the image, the processing chip identifies the camera corresponding to that image as the camera corresponding to the anomaly.

[0084] During the process of the processing chip controlling the camera to shoot at a preset frame rate, when the camera is directly connected to the processing chip, the processing chip directly sends a shooting command to the camera, which controls the camera to shoot at the preset frame rate. When the camera is connected to the processing chip through a path connected by an interface switch, the processing chip sends the aforementioned shooting command to the interface switch. This shooting command also carries an identifier for the camera corresponding to the abnormal situation. The interface switch controls the corresponding camera to shoot at the preset frame rate based on the shooting command. When the camera is connected to this processing chip through another processing chip, the processing chip performing the spatial modeling task sends the aforementioned shooting command to the corresponding processing chip, which then sends the aforementioned shooting command to the corresponding camera. Alternatively, the corresponding processing chip, based on the correspondence between the interface switch and the camera, sends the aforementioned shooting command to the interface switch connected to the corresponding camera, and the interface switch controls the corresponding camera to shoot at the preset frame rate.

[0085] In some embodiments, when the processing chip performing the spatial modeling task detects an abnormal situation, it controls the cameras in the modeling camera group to stop polling exposure, while retaining the camera corresponding to the abnormal situation to continue shooting the environment where the abnormal situation occurs, so as to save the resources consumed in shooting and processing the captured images.

[0086] The above describes one possible implementation whereby the first processing chip, upon detecting an anomaly based on images captured by cameras in the modeling camera group, controls the camera corresponding to the anomaly to capture images at a preset frame rate via a corresponding interface switch. The first processing chip is AP1. By controlling the corresponding camera to continuously capture images when an anomaly is detected, continuous observation of the anomaly can be achieved, enhancing the security of the virtual reality device. In some embodiments, the first processing chip is further configured to, upon detecting a change in the camera corresponding to the anomaly based on images captured by cameras in the modeling camera group, control the corresponding interface switch to switch the capturing camera to the camera corresponding to the anomaly, ensuring continuous observation of the anomaly and further improving the security of the virtual reality device.

[0087] In some embodiments, the modeling camera group includes at least one master camera and at least one slave camera corresponding to each master camera. When the processing chip performing the spatial modeling task detects an abnormal situation, if the camera corresponding to the abnormal situation is a master camera, the processing chip controls the master camera to shoot at a preset frame rate. The slave camera corresponding to the master camera is used to respond to the start of shooting by the master camera corresponding to the slave camera and shoot synchronously with the master camera. This improves the synchronization rate of the cameras shooting synchronously, resulting in higher accuracy in spatial modeling based on the images captured by the master and slave cameras. The master and slave cameras are preset, and the correspondence between them is stored in the processing chip and the camera. The preset frame rate can be 30 frames per second, etc., and this embodiment does not limit this. The synchronous shooting method described above is also called the master-slave exposure method. Figure 4 This is a schematic diagram of a master-slave exposure method provided in an embodiment of this application, as shown below. Figure 4 As shown, during the process of controlling the main camera and the slave camera to shoot synchronously, the processing chip that performs the spatial modeling task sends a shooting command to the main camera. After receiving the shooting command, the main camera sends a frame synchronization signal to the corresponding slave camera so that the slave camera can sense that the main camera has started shooting through the frame synchronization signal and shoot synchronously with the main camera.

[0088] In some embodiments, whether the camera starts shooting in the master-slave exposure mode is related to the algorithm running on the processing chip. For example, if the processing chip detects an abnormal situation based on the algorithm, it will actively send a shooting command (open command) to the corresponding camera to control the corresponding camera to start shooting. If the camera is the master camera, the corresponding slave camera will also shoot synchronously.

[0089] In some embodiments, during the process of determining the main camera and the corresponding slave cameras, the processing chip divides the multiple cameras in the modeled camera group into a preset number of camera groups based on their positions. Each camera group includes at least two cameras. For each camera group, the processing chip determines any one camera in that camera group as the main camera and the remaining cameras in that camera group as slave cameras. For example, as... Figure 2 As shown, cameras 1, 2, 11 and 12 form a camera group, with camera 1 as the master camera and cameras 2, 11 and 12 as slave cameras; cameras 3, 4, 9 and 10 form a camera group, with camera 3 as the master camera and cameras 4, 9 and 10 as slave cameras; cameras 5, 6, 7 and 8 form a camera group, with camera 5 as the master camera and cameras 6, 7 and 8 as slave cameras.

[0090] The above process illustrates the master-slave exposure method using the example where both the master camera and the slave camera are cameras in a modeling camera group. In some embodiments, the master camera and its corresponding slave camera are cameras in different camera groups. For example, the master camera is a camera in a modeling camera group, and the corresponding slave camera is a camera in a perspective camera group. Alternatively, the master camera and its corresponding slave camera may both be cameras in other camera groups. For example, the master camera and its corresponding slave camera may both be cameras in a spatial audio camera group. This application does not limit this aspect.

[0091] The above process illustrates the synchronous shooting process of cameras using a master-slave exposure method as an example. In some embodiments, the virtual reality device uses a slave exposure method to control multiple cameras to shoot synchronously. Accordingly, such as Figure 5 As shown, the processing chip simultaneously sends shooting commands to multiple cameras, and these cameras begin shooting upon receiving the shooting commands. This embodiment of the application does not limit the scope of the invention. The above method, where the processing chip controls the cameras to shoot synchronously, improves the flexibility of synchronous shooting compared to pre-setting cameras to shoot synchronously.

[0092] It should be noted that, due to the limited number of cameras that can be connected to the camera interface on the processing chip, multiple cameras shooting simultaneously are not connected to the same interface switch.

[0093] The above process is illustrated by taking the example of a processing chip performing a spatial modeling task identifying a camera corresponding to an abnormal situation. In some embodiments, after obtaining the above-mentioned fused image, the processing chip performing the rendering task identifies the camera corresponding to the above-mentioned abnormal situation through the fused image and sends a shooting command to the camera corresponding to the abnormal situation. This application embodiment does not limit this.

[0094] In some embodiments, when an interface switch is connected to at least two cameras, the virtual reality device employs a time-sharing strategy to invoke the at least two cameras connected to the interface switch. Accordingly, at any given time, only one of the at least two cameras connected to the interface switch can receive a shooting command sent by the processing chip through the path connected by the interface switch, and then return the captured image to the processing chip through the same path. The shooting command sent by the processing chip to the interface switch carries a camera identifier, which is used to control the interface switch to connect the path between the processing chip and the camera corresponding to the identifier, enabling communication between the camera and the processing chip. Figure 6 This is a schematic diagram of an interface switch provided in an embodiment of this application, as shown below. Figure 6 As shown, interface switch 601 is connected to processing chip 602 via one MIPI data line, referred to as the first line. Interface switch 601 is also connected to camera 603 and camera 604 via two MIPI data lines, referred to as the second line. Figure 6 As shown, at any given time, the interface switch can only connect the processing chip 602 to one camera. That is, only one second line can be connected to the processing chip 602 via the interface switch 601. Correspondingly, the camera corresponding to this second line receives the shooting command sent by the processing chip 602 through the path connected by the interface switch 601 and returns the captured image to the processing chip 602. Taking the first moment when the interface switch connects the path between camera 603 and the processing chip 602 as an example, at the second moment, the processing chip 602 sends a shooting command to the interface switch 601, instructing the camera 604 to take a picture. According to this shooting command, the interface switch 601 connects the second line corresponding to camera 604 to the aforementioned first line, thus connecting the path between camera 604 and the processing chip 602. This allows camera 604 to receive the shooting command sent by the processing chip 602 and return the captured image to the processing chip 602. During image transmission, the MIPI data line uses the CSI (Camera Serial Interface) MIPI transmission protocol to transmit the images captured by the camera.

[0095] In the aforementioned device, through the interface switch, one camera interface on the processing chip can be connected to at least two cameras. The interface switch expands the number of cameras that the processing chip can connect to, thereby increasing the number of cameras and expanding the spatial modeling image acquisition range even when the number of camera interfaces on the processing chip is limited.

[0096] (2) Gesture Interaction Task

[0097] The following section describes the distribution of cameras in the gesture camera group and the connection between the cameras and the processing chip, using the gesture interaction task as an example. The cameras in the gesture camera group are used to capture images of the preset gesture area.

[0098] In some embodiments, such as Figure 1 and Figure 7 As shown above, Figure 2 The gesture camera group shown has two cameras located at the ends of the wearable pendant, used to capture the area below the front of the pendant, where users can interact with the virtual reality device through specified gestures.

[0099] In some embodiments, the cameras in the gesture camera group are connected to a processing chip that performs gesture interaction tasks, or the cameras in the gesture camera group are connected to a processing chip that does not perform gesture interaction tasks. Figure 2 As shown, both cameras 13 and 14 are connected to AP2. Of course, the above... Figure 2 The connection method between the camera and the processing chip shown is only an example. In some embodiments, all cameras in the gesture camera group are connected to AP1, or some are connected to AP1 and some are connected to AP2. This application does not limit this.

[0100] In some embodiments, when the camera in the gesture camera group is connected to a processing chip that performs gesture interaction tasks, the camera in the gesture camera group captures an image of a preset gesture area, and sends the image to the processing chip connected to the camera. Upon receiving the image, the processing chip performs a gesture interaction task based on the image, that is, it recognizes the gesture in the image and obtains gesture information, which indicates the instruction indicated by the gesture. For example, such as... Figure 2 As shown, camera 13 is the camera in the gesture camera group, and camera 13 is connected to AP2. After camera 13 captures an image of the preset gesture area, it sends the image to AP2, which then performs the gesture interaction task based on the image. During the process of the processing chip recognizing the gesture in the image, the processing chip uses a pre-set gesture recognition algorithm to obtain the corresponding gesture information.

[0101] In some embodiments, when the camera in the gesture camera group is connected to a processing chip that does not perform gesture interaction tasks, after the camera in the gesture camera group captures an image, it sends the captured image to the processing chip connected to the camera. The processing chip then forwards the image to the processing chip that performs the gesture interaction tasks, so that the processing chip that performs the gesture interaction tasks can perform the gesture interaction tasks based on the image.

[0102] In some embodiments, the virtual reality device sequentially executes a gesture interaction task and a rendering task to obtain a virtual reality image, which is used to display the execution process of the command indicated by the gesture in the image. When the processing chip executing the gesture interaction task and the processing chip executing the rendering task are different processing chips, the processing chip executing the gesture interaction task completes the gesture interaction task based on the image sent by the camera and obtains the execution result, i.e., the gesture information. The processing chip executing the gesture interaction task sends the execution result to the processing chip executing the rendering task, which then renders the pre-prepared image based on the execution result to obtain the virtual reality image. For example, as... Figure 2 As shown, AP2 performs a gesture interaction task based on the image captured by camera 13 or camera 14, obtains gesture information, and sends the gesture information to AP1. AP1 then renders a pre-prepared image based on the gesture information to obtain a virtual reality image, or AP1 obtains the virtual reality image corresponding to the gesture information. The pre-prepared image is a virtual reality image obtained through a spatial modeling task or a perspective task, or it may be an image pre-stored in AP1 or received by AP1 via a network. This embodiment does not limit the specific details.

[0103] In some embodiments, the processing chip that performs the rendering task is connected to the display device. After rendering a pre-prepared image to obtain a virtual reality image, the processing chip that performs the rendering task sends the virtual reality image to the display device so that the display device can display the virtual reality image.

[0104] In some embodiments, multiple cameras in the gesture camera group are connected to the processing chip through pathways connected by interface switches. Correspondingly, the processing chip is connected to multiple interface switches, and each interface switch is connected to at least one camera in the gesture camera group. This connection method is similar to the connection method between the camera and the processing chip in the modeling camera group, and will not be described again in the embodiments of this application. For example, as... Figure 2 As shown, AP2 is connected to interface switch 1 and interface switch 2. Interface switch 1 is also connected to camera 13, and interface switch 2 is also connected to camera 14.

[0105] The above describes a possible implementation of multiple processing chips, including a third processing chip, which is used to perform gesture interaction tasks in virtual reality display tasks. The third processing chip is connected to a gesture camera group among multiple cameras and to multiple interface switches. Each interface switch is connected to at least one camera in the gesture camera group. The interface switches are used to control the cameras connected to the interface switches to take pictures at different times. The third processing chip is AP2.

[0106] In some embodiments, Figure 2In the virtual reality device shown, cameras 1 and 13 are mounted at the same end of the wearable pendant, and cameras 12 and 14 are also mounted at the same end of the wearable pendant. That is, cameras 1 and 13 share the same position, and cameras 12 and 14 share the same position. Accordingly, cameras 1 and 13 have the same shooting range, and cameras 12 and 14 have the same shooting range. Therefore, images captured by cameras 1 and 12 can be used not only for spatial modeling tasks but also for gesture interaction tasks, and images captured by cameras 13 and 14 can be used not only for gesture interaction tasks but also for camera modeling tasks. In some embodiments, the virtual reality device retains cameras 1 and 12, or cameras 13 and 14, on the wearable pendant, so that the retained cameras belong to both the spatial modeling camera group and the gesture interaction camera group. That is, spatial modeling tasks and gesture interaction tasks share the same portion of the cameras, which can save camera costs.

[0107] The following describes the workflow of the virtual reality device during the execution of gesture interaction tasks.

[0108] In some embodiments, in response to a gesture interaction task being triggered, the processing chip sends a shooting command to a camera in the gesture camera group, causing that camera to begin shooting. The processing chip may be connected to a camera in the gesture camera group, or it may be unconnected. The processing chip has a gesture interaction-related application installed on it, allowing the user to trigger the gesture interaction task. When the processing chip is connected to a camera in the gesture camera group, it directly sends a shooting command to the corresponding camera in response to the gesture interaction task being triggered. When the processing chip is unconnected to a camera in the gesture camera group, it sends a shooting command to the processing chip corresponding to that camera, which then forwards the shooting command to the corresponding camera.

[0109] In some embodiments, when the camera in the gesture camera group is connected to the processing chip through the path connected by the interface switch, the transmission process of the shooting command is the same as the relevant content of the modeling camera group described above, and will not be repeated here in the embodiments of this application.

[0110] (3) Spatial audio task

[0111] The following section describes the distribution of cameras in the spatial audio camera group and the connection between the cameras and the processing chip, using the spatial audio task as an example. The cameras in the spatial audio camera group are used to capture images of the area where the human ear is located.

[0112] In some embodiments, the camera in the spatial audio camera group is an infrared camera. Accordingly, a marker is installed on the display device at a position corresponding to the human ear, and the infrared camera is used to photograph the marker to obtain an image corresponding to the area where the human ear is located.

[0113] In some embodiments, such as Figure 1 and Figure 7 As shown above, Figure 2 The two cameras in the spatial audio camera group shown are located on the inner ring of the wearable pendant and are used to capture images of the area above the wearable pendant where the ear is located.

[0114] In some embodiments, the cameras in the spatial audio camera group are connected to a processing chip that performs spatial audio tasks, or the cameras in the spatial audio camera group are connected to a processing chip that does not perform spatial audio tasks. Figure 2 As shown, both cameras 15 and 16 are connected to AP1. Of course, the above... Figure 2 The connection method between the camera and the processing chip shown is only an example. All cameras in the above spatial audio camera group are connected to AP2, or some are connected to AP1 and some are connected to AP2. This application embodiment does not limit this.

[0115] In some embodiments, when the cameras in a spatial audio camera array are connected to a processing chip that performs spatial audio tasks, the cameras in the spatial audio camera array capture images of the area where the human ear is located, obtain images, and send the images to the processing chip connected to the cameras. Upon receiving the images, the processing chip performs spatial audio tasks based on the images, that is, identifies the location information of the human ear from the images. During the execution of the spatial audio task, the processing chip uses a pre-set spatial audio algorithm to process the images to obtain the location information of the human ear.

[0116] In some embodiments, when a camera in a spatial audio camera group is connected to a processing chip that does not perform spatial audio tasks, after a camera in the spatial audio camera group captures an image, it sends the captured image to the processing chip connected to that camera. The processing chip then forwards the image to the processing chip that performs the spatial audio task, so that the processing chip performing the spatial audio task can perform the spatial audio task based on the image. For example, such as... Figure 2 As shown, camera 15 is a camera in the spatial audio camera group, and camera 15 is not connected to AP2. After camera 15 takes a picture of the area where the human ear is located and obtains an image, it sends the image to AP1. AP1 sends the image to AP2, and AP2 performs spatial audio tasks based on the image.

[0117] In some embodiments, the wearable pendant further includes at least one acoustic cavity, each cavity including at least one speaker. The virtual reality device sequentially performs a spatial audio task and a rendering task to obtain audio for playback through the acoustic cavity. When the processing chip performing the spatial audio task and the processing chip performing the rendering task are different processing chips, the processing chip performing the spatial audio task completes the spatial audio task based on the image sent by the camera and obtains the execution result, i.e., the position information of the human ear. The processing chip performing the spatial audio task sends the execution result to the processing chip performing the rendering task, which then renders the pre-prepared audio based on the execution result, giving the audio a spatial feel, thus obtaining the audio for playback through the acoustic cavity. For example, as... Figure 2 As shown, AP2 performs a spatial audio task based on images captured by camera 15 or camera 16, obtains the position information of the human ear, and sends this position information to AP1. AP1 then renders pre-prepared audio based on the human ear position information to obtain audio for playback through the acoustic cavity. The pre-prepared audio is either pre-stored in AP1 or received by AP1 via the network; this embodiment does not limit the specific audio. During the rendering process of the pre-prepared audio, AP1 adjusts the positions of sound elements in the audio based on the human ear position information to obtain the audio for playback through the acoustic cavity.

[0118] In some embodiments, the processing chip that performs the rendering task is connected to the acoustic cavity. After rendering the pre-prepared audio to obtain the audio to be played through the acoustic cavity, the processing chip that performs the rendering task sends the audio to the acoustic cavity so that the acoustic cavity plays the audio.

[0119] In some embodiments, multiple cameras in a spatial audio camera group are connected to a processing chip via pathways connected by interface switches. Correspondingly, the processing chip is connected to multiple interface switches, and each interface switch is connected to at least one camera in the spatial audio camera group. This connection method is similar to the connection method between cameras and the processing chip in a modeling camera group, and will not be described again in the embodiments of this application. For example, as... Figure 2 As shown, AP 1 is connected to interface switch 7 and interface switch 8. Interface switch 7 is also connected to camera 15, and interface switch 8 is also connected to camera 16.

[0120] The above describes a possible implementation of multiple processing chips, including a second processing chip, which performs spatial audio tasks in virtual reality display. The second processing chip is connected to a spatial audio camera group among multiple cameras. It is also connected to multiple interface switches, each interface switch being connected to at least one camera in the spatial audio camera group. The interface switches control the cameras connected to them to capture images at different times. In this implementation, the second processing chip is AP2. Introducing spatial audio functionality into a virtual reality device enables the device to play audio with a sense of space, improving the user experience.

[0121] The following describes the workflow of the virtual reality device during the execution of the spatial audio task.

[0122] In some embodiments, in response to the triggering of a spatial audio task, the processing chip sends a shooting command to a camera in the spatial audio camera group, causing the camera to begin shooting. The processing chip may be connected to a camera in the spatial audio camera group, or it may be unconnected. The processing chip has a spatial audio-related application installed, allowing the user to trigger the spatial audio task. When the processing chip is connected to a camera in the spatial audio camera group, it directly sends a shooting command to the corresponding camera in response to the triggering of the spatial audio task. When the processing chip is unconnected to a camera in the spatial audio camera group, it sends a shooting command to the processing chip corresponding to that camera, which then forwards the shooting command to the corresponding camera.

[0123] In some embodiments, when the camera in the spatial audio camera group is connected to the processing chip through the path connected by the interface switch, the transmission process of the shooting command is the same as the relevant content of the modeling camera group described above, and will not be repeated here in the embodiments of this application.

[0124] (4) Perspective Task

[0125] The following section explains the distribution of cameras in the perspective camera group and the connection between the cameras and the processing chip, using the perspective task as an example. The cameras in the perspective camera group are used to photograph the area within the field of view corresponding to the human eye.

[0126] In some embodiments, such as Figure 1 As shown, the cameras in the perspective camera group are mounted on the display device at a position corresponding to the human eye, so that the cameras in the perspective camera group can more accurately simulate the environment that the human eye can see, thereby improving the user experience.

[0127] In some embodiments, the cameras in the perspective camera group are connected to a processing chip that performs perspective tasks, or the cameras in the perspective camera group are connected to a processing chip that does not perform perspective tasks. Figure 2 As shown, both cameras 17 and 18 are connected to AP1. Of course, the above... Figure 2 The connection method between the camera and the processing chip shown is only an example. In some embodiments, all cameras in the above-mentioned perspective camera group are connected to AP2, or some are connected to AP1 and some are connected to AP2. This application embodiment does not limit this.

[0128] In some embodiments, when the cameras in the perspective camera group are connected to a processing chip that performs perspective tasks, the cameras in the perspective camera group capture images of the area within the field of view corresponding to the human eye, obtain images, and send the images to the processing chip connected to the cameras. After receiving the images, the processing chip performs perspective tasks based on the images to obtain a fused image corresponding to the area within the field of view corresponding to the human eye. During the process of the processing chip performing perspective tasks, the processing chip uses a pre-set perspective algorithm to process the images to model the environment observed by the human eye and obtain the fused image.

[0129] In some embodiments, when a camera in a perspective camera group is connected to a processing chip that does not perform perspective tasks, after the camera in the perspective camera group captures an image, it sends the captured image to the processing chip connected to that camera. The processing chip then forwards the image to the processing chip that performs the perspective task, so that the processing chip performing the perspective task can perform the perspective task based on the image. For example, such as... Figure 2 As shown, camera 17 is a camera in the perspective camera group. Camera 17 is connected to AP1 but not to AP2. After camera 17 takes a picture of the area within the field of view corresponding to the human eye, it sends the image to AP1. AP1 sends the image to AP2, and AP2 performs the perspective task based on the image.

[0130] In some embodiments, the virtual reality device sequentially performs perspective and rendering tasks to obtain the aforementioned virtual reality image. This process is similar to the related content in the description of the modeling task above, for example, as... Figure 2 As shown, AP2 performs a perspective task based on the image captured by camera 17 or camera 18, obtains a fused image, and sends the fused image to AP1. AP1 renders the fused image to obtain a virtual reality image.

[0131] In some embodiments, the processing chip that performs the rendering task is connected to the display device. After rendering the fused image to obtain a virtual reality image, the processing chip that performs the rendering task sends the virtual reality image to the display device so that the display device can display the virtual reality image.

[0132] In some embodiments, multiple cameras in the perspective camera group are connected to the processing chip via pathways connected by interface switches. Correspondingly, the processing chip is connected to multiple interface switches, and each interface switch is connected to at least one camera in the perspective camera group. This connection method is similar to the connection method between cameras and the processing chip in the modeling camera group, and will not be described again in the embodiments of this application. In some embodiments, the cameras in the perspective camera group are directly connected to the processing chip, for example, as shown in the example... Figure 2 As shown, AP1 is connected to cameras 17 and 18. Compared with the solution of connecting AP1 to cameras 17 and 18 through an interface switch, this solution enables cameras 17 and 18 to shoot synchronously at any time, ensuring perspective effect and improving user experience.

[0133] The above describes a possible implementation of multiple processing chips, including a fourth processing chip, which performs perspective tasks in virtual reality display tasks and is connected to a perspective camera group among multiple cameras. The fourth processing chip is AP2.

[0134] The following describes the workflow of the virtual reality device during the execution of the perspective task.

[0135] In some embodiments, in response to a perspective task being triggered, the processing chip sends a shooting command to a camera in the perspective camera group, causing the camera to begin shooting. The processing chip may be connected to a camera in the perspective camera group, or it may be unconnected to a camera in the perspective camera group. A perspective-related application is installed on the processing chip, allowing the user to trigger the perspective task. This process is similar to the related content in the process of performing spatial modeling tasks described above, and will not be repeated here.

[0136] (5) Eye-tracking task

[0137] The following section explains the distribution of cameras in the tracking camera group and the connection between the cameras and the processing chip, using the eye-tracking task as an example. The cameras in the tracking camera group are used to capture images of the human eye.

[0138] In some embodiments, such as Figure 1 As shown, the cameras in the tracking camera group are mounted on the display device so that the cameras in the tracking camera group can capture images of the human eye.

[0139] In some embodiments, the cameras in the tracking camera group are connected to a processing chip that performs eye-tracking tasks, or the cameras in the tracking camera group are connected to a processing chip that does not perform eye-tracking tasks. Figure 2 As shown, both cameras 19 and 20 are connected to AP1. Of course, the above... Figure 2The connection method between the camera and the processing chip shown is only an example. In some embodiments, all cameras in the above-mentioned tracking camera group are connected to AP2, or some are connected to AP1 and some are connected to AP2. This application embodiment does not limit this.

[0140] In some embodiments, when a camera in a tracking camera group is connected to a processing chip that performs eye-tracking tasks, the camera in the tracking camera group captures an image of the human eye and sends the image to the processing chip connected to the camera. Upon receiving the image, the processing chip performs an eye-tracking task based on the image to obtain the location information of the position the human eye is looking at. During the eye-tracking task, the processing chip uses a pre-set eye-tracking algorithm to process the image to obtain the location information of the position the human eye is looking at.

[0141] In some embodiments, when a camera in a tracking camera group is connected to a processing chip that does not perform eye-tracking tasks, after a camera in the tracking camera group captures an image, it sends the captured image to the processing chip connected to that camera. This processing chip then forwards the image to the processing chip that performs the eye-tracking task, so that the processing chip performing the eye-tracking task can perform the eye-tracking task based on the image. For example, such as... Figure 2 As shown, camera 19 is a camera in the tracking camera group, and camera 19 is connected to AP1 but not to AP2. After camera 19 takes a picture of the human eye and obtains an image, it sends the image to AP1. AP1 sends the image to AP2, and AP2 performs an eye-tracking task based on the image.

[0142] In some embodiments, the virtual reality device sequentially performs eye-tracking and rendering tasks to obtain a virtual reality image. When the processing chip performing the eye-tracking task and the processing chip performing the rendering task are different processing chips, the processing chip performing the eye-tracking task completes the eye-tracking task based on the image sent by the camera and obtains the execution result, i.e., the position information of the location being gazed at. The processing chip performing the eye-tracking task sends the execution result to the processing chip performing the rendering task, which then renders the pre-prepared image based on the execution result to obtain the virtual reality image. For example, as... Figure 2 As shown, AP2 performs an eye-tracking task based on images captured by camera 19 or camera 20, obtains the location information of the position being gazed at, and sends this information to AP1. AP1 then renders a pre-prepared image based on the location information of the position being gazed at, thus obtaining a virtual reality image. The pre-prepared image is a virtual reality image obtained through a spatial modeling task or a perspective task, or it may be an image pre-stored in AP1 or received by AP1 via a network. This embodiment does not limit the specific type of image.

[0143] In some embodiments, the processing chip that performs the rendering task is connected to the display device. After rendering a pre-prepared image to obtain a virtual reality image, the processing chip that performs the rendering task sends the virtual reality image to the display device so that the display device can display the virtual reality image.

[0144] In some embodiments, multiple cameras in the tracking camera group are connected to the processing chip via pathways connected by interface switches. Correspondingly, the processing chip is connected to multiple interface switches, and each interface switch is connected to at least one camera in the tracking camera group. This connection method is similar to the connection method between cameras and the processing chip in the modeling camera group, and will not be described again in the embodiments of this application. In some embodiments, the cameras in the tracking camera group are directly connected to the processing chip, for example, as... Figure 2 As shown, AP1 is connected to cameras 19 and 20. Compared with the solution of connecting AP1 to cameras 19 and 20 through an interface switch, cameras 19 and 20 can shoot synchronously at any time, ensuring the accuracy of eye tracking and improving the user experience.

[0145] The above describes a possible implementation in which multiple processing chips, including a fifth processing chip, are connected to a tracking camera group among multiple cameras, and the fifth processing chip is AP2.

[0146] The following describes the workflow of the virtual reality device during the execution of the eye-tracking task.

[0147] In some embodiments, the processing chip responds to the power-on command of the virtual reality device and sends a shooting command to a camera in the tracking camera group, causing the camera to start shooting. The processing chip may be connected to a camera in the tracking camera group, or it may be unconnected to a camera in the tracking camera group. An eye-tracking application is installed on the processing chip, allowing the virtual reality device to trigger eye-tracking tasks. This process is similar to the related content in the spatial modeling task execution process described above, and will not be repeated here.

[0148] The above content is based on Figure 2 For example, the connection between the camera and the processing chip in a virtual reality device and the data transmission process during task execution are illustrated. Figure 2The connection relationship between the camera and the processing chip shown takes into account the amount of data corresponding to the images captured by the camera, the functions of different processing chips, and the number of communication cycles between the processing chips. This minimizes the number of communication cycles between AP1 and AP2, thereby improving the efficiency of the virtual reality device in performing virtual reality display tasks. For example, Figure 2 In the virtual reality device shown, AP1 primarily performs rendering tasks and runs a user application integrating spatial modeling and spatial audio functions. This application responds to user commands, executes tasks, and displays the results on the display device. AP2 primarily executes demanding algorithms (algorithms requiring more than 5 TOPS of NPU computing power or more than 50 GFLOPS of GPU computing power), such as spatial modeling and spatial audio algorithms, bearing the operational burden of these algorithms. It also processes images from AP1 and sends the processing results back to AP1. Furthermore, the display device is connected to AP1; the processing results from AP2 are transmitted to AP1 before being displayed on the device. Since AP2 is the processing chip for spatial modeling tasks, it is connected to as many cameras as possible in the modeling camera group, enabling AP2 to directly receive images captured by these cameras and improve task execution efficiency.

[0149] It should be noted that the above content is based on Figure 2 In this example, AP1 is a processing chip that performs rendering tasks, and AP2 is a processing chip that performs spatial modeling tasks, spatial audio tasks, gesture interaction tasks, and eye-tracking tasks. This application does not limit the tasks performed by the processing chips; for example, in some embodiments... Figure 2 AP1 in the system performs tasks such as gesture interaction and eye tracking.

[0150] The following is based on Figure 2 Taking this as an example, the interaction process between processing chips will be explained. Figure 2 As shown, AP1 and AP2 are connected via a PCIe (Peripheral Component Interconnect express, a high-speed serial computer expansion bus standard) bus, and correspondingly, as... Figure 8As shown, both AP1 and AP2 include PCIe modules connected to the PCIe bus. These modules receive data from the PCIe bus and also send data to other processing chips via the PCIe bus. After a handshake connection, the PCIe modules in AP1 and AP2 can interact with each other. Taking AP1 as an example, a user application is installed on a processing chip. AP1 receives user commands through this application. Data transmission between the PCIe module and the application in AP1 is achieved through the libpic library. The transmitted data includes, but is not limited to, handshake information between multiple processing chips, images sent by other processing chips, or processing results. Correspondingly, the PCIe module receives data from AP2 and sends it to the corresponding application in AP1 through the libpic library. AP1 also has a kernel BSP driver installed. The application connects to the BSP (Board Support Package) via the ioctl interface. The BSP stores camera authentication information, including the corresponding camera's identifier. When AP1 receives images from a camera, it receives the images from the authenticated camera through the BSP. Correspondingly, when AP1 sends a shooting command to the camera, it also sends the shooting command to the corresponding certified camera through the BSP. Here, a certified camera refers to a camera whose corresponding certification information is stored in the BSP.

[0151] Taking a single data interaction process as an example, the application on AP1 receives images captured by the camera connected to AP1 via the BSP and ioctl interface. Using the libpic library, it sends these images to the PCIe module in AP1. The PCIe module in AP1 then sends the images to the PCIe module in AP2 via the PCIe bus. The PCIe module in AP2 then sends the images to the application in AP2 via the libpic library. The BSP on AP2 also receives images captured by the camera connected to AP2, allowing the application in AP2 to process and integrate the received images uniformly. The application on AP1 receives user commands and renders images or audio according to these commands. The application on AP2 executes spatial modeling algorithms, spatial audio algorithms, gesture interaction algorithms, and eye-tracking algorithms, providing background services for the application on AP1. For example, in response to a user's trigger operation for the spatial audio function, AP1 acquires images simultaneously captured by cameras 15 and 16 and transmits these images to AP2 via the PCIe bus. The application on AP2 performs algorithmic and image fusion calculations on the image and sends the calculation results back to AP1 via the PCIe bus. The spatial audio application on AP1 then outputs audio based on the calculated structure.

[0152] In some embodiments, the multiple processing chips of a virtual reality device include a master chip and slave chips corresponding to the master chip. The master chip is the control terminal, and the slave chip is the controlled terminal. The master chip is used to control the slave chip to start running. For example... Figure 2 In this setup, the master chip is AP1, and the slave chip is AP2. AP1 responds to the power-on command, starts running, and sends a run command to AP2 to control AP2 to start running. The master and slave chips are pre-configured within the virtual reality device.

[0153] As can be seen from the above, the data received by the processing chip needs to be processed and integrated by the application program on the processing chip. The framework for the application program to process and integrate the data is as follows: Figure 9 As shown, the application receives data through the PCIe module or BSP. The application decapsulates the data received through the PCIe module to obtain images from multiple cameras. The application uses corresponding algorithms to process these images and outputs the processing results to the display device or sound cavity.

[0154] It should be noted that the above description is based on the example of a virtual reality device that is a separate device. In some embodiments, the virtual reality device is a non-separate device, and this application does not limit this.

[0155] Furthermore, the above description of the processing chip for handling rendering tasks refers to a plurality of processing chips including a sixth processing chip. This sixth processing chip is used to receive the processing result obtained by any one of the processing chips and to execute the rendering task in the virtual reality display task. The sixth processing chip is AP1. As can be seen from the above description, at least two of the first to sixth processing chips may be the same chip or different chips; this application embodiment does not limit this. Any one of the plurality of processing chips is also used to send images captured by the cameras in a camera group that is connected to a camera in a camera group whose task does not match the task being processed by that processing chip, to the processing chip corresponding to the task corresponding to that camera group. Any one of the plurality of processing chips is also used to send the processing result to the processing chip corresponding to the next task after completing the task corresponding to that processing chip, so that the plurality of processing chips can collaboratively complete the virtual reality display task, reducing costs.

[0156] The virtual reality device provided in this application solves the problem that a single low-performance SoC cannot connect to multiple cameras by combining dual SoCs with an interface switch. It enables the device to set up multiple cameras and multiple processing chips connected via a bus. Each processing chip is connected to some of the cameras, so that multiple processing chips share and coordinate virtual reality display tasks. This breaks through the high computing power that can only be achieved by high-performance processors. By using multiple low-performance processors to achieve the processing requirements of high computing power algorithms, the performance requirements of a single processing chip are reduced, thereby effectively reducing the overall cost of the VR device. It can also realize functions such as spatial modeling and spatial audio based on low computing power processing chips, improve the security of virtual reality devices, and bring users an immersive experience.

[0157] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules in virtual reality display. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0158] The foregoing description provides an exemplary account of the virtual reality device provided in the embodiments of this application. The following describes a method applied to the aforementioned device. In this method, multiple processing chips, based on images captured by multiple cameras, respectively execute partial tasks within a virtual reality display task to obtain a virtual reality image. The virtual reality image is then transmitted to a display device, causing the display device to display the virtual reality image. The following description, in conjunction with... Figure 10 ,by Figure 2 Taking the virtual reality device shown as an example of performing spatial modeling and rendering tasks, the virtual reality display method applied to this virtual reality device is explained. Figure 10 This is a data interaction diagram of a virtual reality display method provided in an embodiment of this application. For example... Figure 10 As shown, the method includes the following steps.

[0159] 1001. In response to detecting an abnormal situation, AP1 sends a shooting command to AP2, which controls cameras 1, 5, 6, 7, 8 and 12 to take pictures.

[0160] The process by which AP1 detects abnormal conditions is similar to that in the aforementioned virtual reality device, and will not be repeated here. In this embodiment, when AP1 detects an abnormal condition, and the camera corresponding to the abnormal condition is camera 1, 5, 6, 7, 8, or 12, it sends a shooting command to AP2.

[0161] 1002 and AP2 respond to the shooting command and control cameras 1, 5, 6, 7, 8 and 12 to take pictures and obtain images.

[0162] 1003. AP1 responds to the change in the camera corresponding to the abnormal situation and sends a switching command to AP2. This switching command is used to control cameras 2, 3, 4, 9, 10 and 11 to take pictures.

[0163] In this embodiment of the application, when AP1 detects that the camera corresponding to the abnormal situation has changed to camera 2, 3, 4, 9, 10 and 11, it sends a switching command to AP2.

[0164] 1004 and AP2 respond to the switching command and control cameras 2, 3, 4, 9, 10 and 11 to take pictures and obtain images.

[0165] Cameras 2, 4, 9, and 11 are connected to the same interface switch as cameras 5, 6, 7, and 8, respectively, and cannot take pictures simultaneously.

[0166] It should be noted that since cameras 3 and 10 are connected to AP1, after AP1 obtains the images captured by cameras 3 and 10, it sends the images to AP2 so that AP2 can process the images.

[0167] 1005. AP2 uses a spatial modeling algorithm to process the images captured by cameras 1 to 12 to obtain a fused image, and sends the fused image to AP1.

[0168] 1006. AP1 renders the fused image to obtain a virtual reality image, and sends the virtual reality image to the display device.

[0169] 1007. Display devices display virtual reality images.

[0170] The above process is similar to that in virtual reality devices, and will not be repeated here. It should be noted that the above process is illustrated using the example of a virtual reality device detecting that spatial modeling is needed based on images captured by 12 cameras. Since multiple cameras on the same interface switch cannot capture images simultaneously, the virtual reality device will first control some cameras to capture images, and then control the other group of cameras to capture images. Through the above capturing process, 12 images can be obtained. In some embodiments, the processing chip processes some of these images. The processing chip filters the images based on the camera's identifier to obtain the image captured by the corresponding camera. That is, the processing chip is also used to identify the image captured by the camera based on the camera's identifier, which can improve processing efficiency.

[0171] In addition, the above Figure 10The process shown does not involve cameras 17-20. Since the above four cameras do not involve interface switches and are directly connected to the processing chip, AP1 does not need to send switching commands to these four cameras. After acquiring the images captured by these four cameras as needed, it can transmit them to AP2 via the PCIe bus. The application on AP2 receives the images and processes them.

[0172] The virtual reality display method provided in this application distributes virtual reality display tasks to multiple processing chips connected by a bus, enabling multiple processing chips to share and coordinate virtual reality display tasks. This reduces the performance requirements of individual processing chips and effectively lowers costs.

[0173] This application also provides a computer-readable storage medium storing at least one computer program. This computer program is loaded and executed by the processor of a virtual reality device to implement the operations performed by the server in the virtual reality display method of the above embodiments. For example, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0174] This application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. The processor of the virtual reality device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the virtual reality device to perform the virtual reality display method provided in the various optional implementations described above.

[0175] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0176] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A virtual reality device, characterized in that, The virtual reality device includes: A wearable pendant and a display device, wherein the wearable pendant and the display device are connected; The wearable pendant includes multiple cameras and multiple processing chips disposed on the wearable pendant. The multiple processing chips are connected to each other via a bus. Each processing chip is connected to a portion of the multiple cameras. The multiple processing chips are used to perform a portion of the virtual reality display task based on the images captured by the multiple cameras, so as to obtain a virtual reality image and transmit the virtual reality image to the display device. The display device is used to display the virtual reality image.

2. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a first processing chip, which is used to perform the spatial modeling task in the virtual reality display task. The first processing chip is connected to the modeling camera group among the plurality of cameras. The first processing chip is also connected to a plurality of interface switches. Each interface switch is connected to at least one camera in the modeling camera group. The cameras in the modeling camera group are used to capture images of the environment within a corresponding angle range. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

3. The virtual reality device according to claim 2, characterized in that, The multiple cameras in the modeling camera group take pictures of the environment within the corresponding angle range in a preset shooting order.

4. The virtual reality device according to claim 2, characterized in that, The wearable pendant has a ring-shaped structure. The cameras in the modeling camera group are distributed on the outer ring of the wearable pendant according to preset positions, so that the shooting range of the cameras in the modeling camera group covers the angle range corresponding to the wearable pendant.

5. The virtual reality device according to claim 2, characterized in that, The first processing chip is also used to control the camera corresponding to the abnormal situation to shoot at a preset frame rate through the corresponding interface switch when an abnormal situation is detected based on the images captured by the cameras in the modeling camera group.

6. The virtual reality device according to claim 5, characterized in that, The first processing chip is also configured to, when detecting a change in the camera corresponding to the abnormal situation based on the images captured by the cameras in the modeling camera group, control the corresponding interface switch to switch the camera performing the shooting to the camera corresponding to the abnormal situation.

7. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a second processing chip, which is used to execute the spatial audio task in the virtual reality display task. The second processing chip is connected to the spatial audio camera group among the plurality of cameras. The second processing chip is also connected to a plurality of interface switches. Each interface switch is connected to at least one camera in the spatial audio camera group. The cameras in the spatial audio camera group are used to capture images of the human ear. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

8. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a third processing chip, which is used to perform gesture interaction tasks in the virtual reality display task. The third processing chip is connected to the gesture camera group among the plurality of cameras. The third processing chip is also connected to a plurality of interface switches. Each interface switch is connected to at least one camera in the gesture camera group. The cameras in the gesture camera group are used to capture images of a preset gesture area. The interface switches are used to control the cameras connected to the interface switches to capture images at different times.

9. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a fourth processing chip, which is used to perform the perspective task in the virtual reality display task. The fourth processing chip is connected to the perspective camera group among the plurality of cameras, and the cameras in the perspective camera group are used to capture images of the environment within the human eye's field of vision.

10. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a fifth processing chip, which is used to perform the eye-tracking task in the virtual reality display task. The fifth processing chip is connected to the tracking camera group among the plurality of cameras, and the cameras in the tracking camera group are used to capture images of the human eye.

11. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes a sixth processing chip, which is used to receive the processing result obtained by any one of the plurality of processing chips, and to execute the rendering task in the virtual reality display task on the processing result.

12. The virtual reality device according to any one of claims 1-11, characterized in that, Any of the plurality of processing chips is further configured to, when connected to a camera in a camera group whose task does not correspond to that of the processing chip, send images captured by the camera in the camera group to the processing chip corresponding to the task of the camera group.

13. The virtual reality device according to any one of claims 1-11, characterized in that, Any of the plurality of processing chips is further configured to, upon completing the task corresponding to the processing chip, send the processing result to the processing chip corresponding to the next task of the task corresponding to the processing chip.

14. The virtual reality device according to claim 1, characterized in that, The plurality of cameras includes at least one main camera and at least one slave camera corresponding to each main camera; The slave camera is used to respond to the main camera corresponding to the slave camera starting to shoot, and to shoot synchronously with the main camera.

15. The virtual reality device according to claim 14, characterized in that, Multiple cameras shooting simultaneously are not connected to the same interface switch.

16. The virtual reality device according to claim 1, characterized in that, The plurality of processing chips includes at least one main processing chip and a slave processing chip corresponding to each main processing chip; The main processing chip is used to control the slave processing chip to start running.

17. The virtual reality device according to claim 1, characterized in that, The processing chip is also used to identify images captured by the camera based on the camera's identifier.

18. A virtual reality display method, characterized in that, The virtual reality device includes a wearable accessory and a display device. The wearable accessory and the display device are connected. The wearable accessory includes multiple cameras and multiple processing chips disposed on the outer ring of the wearable accessory. The multiple processing chips are connected to each other via a bus, and each processing chip is connected to a portion of the multiple cameras. The method includes: The multiple processing chips perform a portion of the virtual reality display task based on the images captured by the multiple cameras, in order to obtain virtual reality images and transmit the virtual reality images to the display device; The display device displays the virtual reality image.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one computer program for performing the method of claim 18.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 18.

Images