Data transmission method and device, head-mounted display equipment and medium

Abstract

The embodiment of the invention discloses a data transmission method and device, head-mounted display equipment and a medium, applied to the head-mounted display equipment, the method comprises the following steps: receiving a data acquisition request sent by a target application; wherein the data acquisition request is used for acquiring target data of the head-mounted display equipment, and the target data comprises camera data and sensor data; and in response to the data acquisition request, controlling the target application to acquire the target data through a camera service of the head-mounted display device.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted display technology, and more specifically, to a data transmission method, a data transmission apparatus, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] Currently, head-mounted display devices, such as Mixed Reality (MR) products and Augmented Reality (AR) products, typically use the Android operating system. Applications need to obtain camera data and sensor data through different services. For example, applications obtain camera data through the Camera Service and sensor data through the Sensor Service. However, since the Sensor Service usually has a large delay when obtaining sensor data, this reduces the synchronization accuracy of camera data and sensor data. Summary of the Invention

[0003] The purpose of this disclosure is to provide a data transmission method, apparatus, head-mounted display device, and medium.

[0004] According to a first aspect of the present disclosure, a data transmission method is provided, applied to a head-mounted display device, the method comprising:

[0005] Receive a data acquisition request sent by a target application; wherein the data acquisition request is used to acquire target data of the head-mounted display device, and the target data includes camera data and sensor data;

[0006] In response to the data acquisition request, the target application is controlled to acquire the target data through the camera service of the head-mounted display device.

[0007] Optionally, in response to the data acquisition request, controlling the target application to acquire the target data through the camera service of the head-mounted display device includes:

[0008] In response to the data acquisition request, the target application is controlled to acquire the camera data through the camera service, and the target application is controlled to acquire the sensor data through the sensor data callback interface of the camera service.

[0009] Optionally, the step of controlling the target application to acquire the camera data through the camera service and controlling the target application to acquire the sensor data through the sensor data callback interface of the camera service in response to the data acquisition request includes:

[0010] In response to the data acquisition request, the target application is controlled to acquire the camera data through the camera service and the camera hardware abstraction layer of the head-mounted display device, and the target application is controlled to acquire the sensor data through the sensor data callback interface of the camera service and the low-latency interface of the camera hardware abstraction layer.

[0011] Optionally, the camera hardware abstraction layer acquires the camera data through the image signal processor of the head-mounted display device, and the low-latency interface of the camera hardware abstraction layer acquires the sensor data through the digital signal processor of the head-mounted display device.

[0012] Optionally, the method further includes:

[0013] The target data read is encoded using the camera service to obtain the encoded target data;

[0014] The encoded target data is transmitted to the target application via the camera service.

[0015] Optionally, after responding to the data acquisition request and controlling the target application to acquire the target data through the camera service of the head-mounted display device, the method further includes:

[0016] Control the target application to perform the target operation based on the target data.

[0017] Optionally, the target operation includes head tracking and gesture recognition.

[0018] According to a second aspect of the present disclosure, a data transmission apparatus is provided for use in a head-mounted display device, the apparatus comprising:

[0019] A receiving module is used to receive a data acquisition request sent by a target application; wherein the data acquisition request is used to acquire target data of the head-mounted display device, and the target data includes camera data and sensor data;

[0020] The acquisition module is used to respond to the data acquisition request and control the target application to acquire the target data through the camera service of the head-mounted display device.

[0021] According to a third aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device comprising:

[0022] Memory is used to store executable computer instructions;

[0023] A processor, configured to execute the data transmission method according to the first aspect above, under the control of the executable computer instructions.

[0024] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon, which, when executed by a processor, perform the data transmission method described in the first aspect above.

[0025] One beneficial effect of this disclosure is that, after receiving a data acquisition request from a target application for obtaining camera and sensor data from the head-mounted display device, the head-mounted display device can respond to the request and control the target application to simultaneously acquire camera and sensor data through the head-mounted display device's camera service. In other words, it designs a new software architecture for accessing camera and sensor data within the Android operating system. The target application can access both camera and sensor data simultaneously through the camera service alone, optimizing the code structure, reducing code complexity, decreasing call latency, and improving the synchronization accuracy of camera and sensor data.

[0026] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0028] Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted display device according to an embodiment of the present disclosure;

[0029] Figure 2 This is a flowchart illustrating a data transmission method according to an embodiment of the present disclosure;

[0030] Figure 3 This is a software architecture for accessing camera data and sensor data within the Android operating system, provided according to embodiments of this disclosure;

[0031] Figure 4 This is a schematic block diagram of a data transmission apparatus according to an embodiment of the present disclosure;

[0032] Figure 5 This is a schematic block diagram of a head-mounted display device according to an embodiment of the present disclosure. Detailed Implementation

[0033] Figure 1 This is a schematic diagram of the hardware configuration that can be used to implement one embodiment of a head-mounted display device.

[0034] In one embodiment, such as Figure 1As shown, the head-mounted display device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a microphone 1700, and a speaker 1800, etc. The processor 1100 may include, but is not limited to, a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces), parallel bus interfaces, etc. The communication device 1400 may be capable of wired or wireless communication, for example. The display device 1500 may be, for example, a liquid crystal display (LCD), an LED display, a touch screen, etc. The input device 1600 may include, for example, a touch screen. The microphone 1700 may be used to input voice information. The speaker 1800 may be used to output voice information.

[0035] The head-mounted display device 1000 may be an AR (Augmented Reality) device or an MR (Mixed Reality) video perspective device, etc., and this disclosure does not limit it. Figure 1 The components shown are merely illustrative. The head-mounted display device 1000 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The head-mounted display device 1000 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0036] In this embodiment, the memory 1200 of the head-mounted display device 1000 is used to store program instructions that control the processor 1100 to perform data transmission methods. Those skilled in the art can design these instructions based on the scheme disclosed herein. How the instructions control the processor is well-known in the art and will not be described in detail here.

[0037] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0038] <Method Implementation>

[0039] Figure 2 This disclosure illustrates a data transmission method according to an embodiment of the present disclosure, which can be provided by... Figure 1 The illustrated head-mounted display device can be an MR video see-through device or an AR device. Figure 2As shown, the data transmission method of this embodiment may include the following steps S2100 to S2200:

[0040] Step S2100: Receive a data acquisition request sent by the target application.

[0041] The data acquisition request is used to acquire target data from the head-mounted display device, including camera data and sensor data. This data acquisition request can be triggered actively by the target application or by the user through the target application.

[0042] The target application is any application currently running on the head-mounted display device, including but not limited to games and video applications.

[0043] Typically, head-mounted display devices are equipped with cameras, and the aforementioned camera data refers to image data captured by the cameras of the head-mounted display device. Head-mounted display devices are also equipped with inertial measurement units (IMUs), and the aforementioned sensor data refers to IMU data captured by the IMUs of the head-mounted display device.

[0044] After executing S2100 to receive the data acquisition request sent by the target application, proceed to:

[0045] Step S2200: In response to the data acquisition request, control the target application to acquire the target data through the camera service of the head-mounted display device.

[0046] In this embodiment, the target application can access both camera data and sensor data simultaneously through the Camera Service.

[0047] In an optional embodiment, step S2200, in response to the data acquisition request, controlling the target application to acquire the target data through the camera service of the head-mounted display device, may further include: in response to the data acquisition request, controlling the target application to acquire the camera data through the camera service, and controlling the target application to acquire the sensor data through the sensor data callback interface of the camera service.

[0048] Specifically, the above-mentioned response to the data acquisition request, controlling the target application to acquire the camera data through the camera service, and controlling the target application to acquire the sensor data through the sensor data callback interface of the camera service, may further include: in response to the data acquisition request, controlling the target application to acquire the camera data through the camera service and the camera hardware abstraction layer of the head-mounted display device, and controlling the target application to acquire the sensor data through the sensor data callback interface of the camera service and the low-latency interface of the camera hardware abstraction layer.

[0049] The Camera Hardware Abstraction Layer (Camera HAL) acquires camera data through the Image Signal Processor (ISP) of the head-mounted display device, and the low-latency interface of the Camera Hardware Abstraction Layer of the head-mounted display device acquires sensor data through the Digital Signal Processor (DSP).

[0050] The Hardware Abstraction Layer (HAL) in the Android system encapsulates the underlying hardware drivers and provides a general interface for calling the drivers to the software layer (Framework layer).

[0051] It should be noted that in related technologies, the software architecture for accessing camera data and sensor data in the Android operating system of head-mounted display devices is typically as follows: the target application accesses sensor data through the Sensor Service and the Sensor Hardware Abstraction Layer (Sensor HAL), and the target application accesses camera data through the Camera Service and the Camera Hardware Abstraction Layer.

[0052] In this embodiment, the target application can simultaneously access camera data and sensor data through the camera service and the camera hardware abstraction layer (Camera HAL). In other words, it designs a new software architecture for accessing camera and sensor data within the Android operating system of the head-mounted display device. Instead of using the standard Android operating system flow of sensor service and sensor hardware abstraction layer, it adds interfaces for accessing and calling sensor data to the camera service and camera hardware abstraction layer. Specifically, it adds a sensor data callback interface (IMU Data Callback API) to the camera service and a low-latency sensor interface (Low Latency IMU API) to the camera hardware abstraction layer.

[0053] In practice, the target application receives camera data through the camera service and the camera hardware abstraction layer, and receives sensor data through the sensor data callback interface of the camera service and the low-latency sensor interface of the camera hardware abstraction layer. This optimizes the code structure, reducing code complexity, minimizing call latency, and improving the synchronization accuracy of sensor and camera data.

[0054] According to embodiments of this disclosure, after receiving a data acquisition request from a target application for obtaining camera and sensor data from the head-mounted display device, the head-mounted display device can respond to the request by controlling the target application to simultaneously acquire camera and sensor data through the head-mounted display device's camera service. In other words, it designs a new software architecture for accessing camera and sensor data within the Android operating system. The target application can access both camera and sensor data simultaneously through the camera service alone, optimizing the code structure, reducing code complexity, decreasing call latency, and improving the synchronization accuracy of camera and sensor data.

[0055] In one embodiment, after performing the above step S2200 in response to the data acquisition request and controlling the target application to acquire the target data through the camera service of the head-mounted display device, the data transmission method of this embodiment further includes: controlling the target application to perform a target operation based on the target data.

[0056] The aforementioned target operations include, but are not limited to, head tracking and gesture recognition.

[0057] In one embodiment, the data transmission method of this disclosure further includes the following steps S3100 to S3200:

[0058] Step S3100: Encode the target data read by the camera service to obtain the encoded target data.

[0059] Specifically, after the camera service of the head-mounted display device reads the sensor data of the camera data of the head-mounted display device through the camera hardware abstraction layer, the camera service of the head-mounted display device encodes the read camera data and sensor data respectively to obtain the encoded camera data and encoded sensor data.

[0060] Step S3200: Transmit the encoded target data to the target application via the camera service.

[0061] Specifically, the camera service of the head-mounted display device can send encoded camera data and encoded sensor data to the target application for decoding.

[0062] According to this embodiment, the head-mounted display device can encode camera data and sensor data before transmission, thereby reducing the amount of data transmitted and increasing the data transmission speed.

[0063] <Example>

[0064] The following is an example of a data transmission method, see reference. Figure 3 In this example, the data transmission method includes:

[0065] Step S401: Receive a data acquisition request sent by the target application.

[0066] The data acquisition request is used to acquire target data from the head-mounted display device, including camera data and sensor data.

[0067] In step S402, in response to the data acquisition request, the target application is controlled to acquire camera data output by the image signal processor through the camera service and the camera hardware abstraction layer, and the target application is controlled to acquire sensor data output by the digital signal processor through the sensor data callback interface of the camera service and the low-latency sensor interface of the camera hardware abstraction layer.

[0068] This example demonstrates how a low-latency architecture can be set up within the Android operating system of a head-mounted display device. This ensures that sensor data is transmitted simultaneously with camera data transmission, optimizes the code structure, reduces code complexity, decreases call latency, and improves the synchronization of camera and sensor data.

[0069] <Device Embodiment>

[0070] Figure 4 This is a schematic diagram of a data transmission device according to one embodiment, with reference to... Figure 4 As shown, the data transmission device 400 includes a receiving module 410 and an acquiring module 420.

[0071] The receiving module 410 is used to receive a data acquisition request sent by a target application; wherein the data acquisition request is used to acquire target data of the head-mounted display device, and the target data includes camera data and sensor data;

[0072] The acquisition module 420 is configured to, in response to the data acquisition request, control the target application to acquire the target data through the camera service of the head-mounted display device.

[0073] In one embodiment, the acquisition module 420 is configured to, in response to the data acquisition request, control the target application to acquire the camera data through the camera service, and control the target application to acquire the sensor data through the sensor data callback interface of the camera service.

[0074] In one embodiment, the acquisition module 420 is configured to, in response to the data acquisition request, control the target application to acquire the camera data through the camera service and the camera hardware abstraction layer of the head-mounted display device, and control the target application to acquire the sensor data through the sensor data callback interface of the camera service and the low-latency interface of the camera hardware abstraction layer.

[0075] In one embodiment, the camera hardware abstraction layer acquires camera data through the image signal processor of the head-mounted display device, and the low-latency interface of the camera hardware abstraction layer acquires sensor data through the digital signal processor of the head-mounted display device.

[0076] In one embodiment, the device 400 further includes an encoding module (not shown in the figure), which is used to encode the target data read by the camera service to obtain the encoded target data; and to transmit the encoded target data to the target application by the camera service.

[0077] In one embodiment, the device 400 further includes a control module (not shown) for controlling the target application to perform a target operation based on the target data after the acquisition module 420 controls the target application to acquire the target data through the camera service of the head-mounted display device in response to the data acquisition request.

[0078] In one embodiment, the target operation includes head tracking and gesture recognition.

[0079] According to embodiments of this disclosure, after receiving a data acquisition request from a target application for obtaining camera and sensor data from the head-mounted display device, the head-mounted display device can respond to the request by controlling the target application to simultaneously acquire camera and sensor data through the head-mounted display device's camera service. In other words, it designs a new software architecture for accessing camera and sensor data within the Android operating system. The target application can access both camera and sensor data simultaneously through the camera service alone, optimizing the code structure, reducing code complexity, decreasing call latency, and improving the synchronization accuracy of camera and sensor data.

[0080] <Equipment Example>

[0081] Figure 5 This is a schematic diagram of the hardware structure of a head-mounted display device according to one embodiment. Figure 5 As shown, the head-mounted display device 500 includes a processor 510 and a memory 520.

[0082] The memory 520 can be used to store executable computer instructions.

[0083] The processor 510 can be used to execute the data transmission method according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0084] The head-mounted display device 500 can be as follows: Figure 1 The head-mounted display device 1000 shown may also be a device with other hardware structures, which are not limited here.

[0085] In another embodiment, the head-mounted display device 500 may include the above-mentioned data transmission device 400.

[0086] In one embodiment, each module of the data transmission device 400 can be implemented by the processor 510 running computer instructions stored in the memory 520.

[0087] According to embodiments of this disclosure, after receiving a data acquisition request from a target application for obtaining camera and sensor data from the head-mounted display device, the head-mounted display device can respond to the request by controlling the target application to simultaneously acquire camera and sensor data through the head-mounted display device's camera service. In other words, it designs a new software architecture for accessing camera and sensor data within the Android operating system. The target application can access both camera and sensor data simultaneously through the camera service alone, optimizing the code structure, reducing code complexity, decreasing call latency, and improving the synchronization accuracy of camera and sensor data.

[0088] Computer-readable storage media

[0089] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the data transmission method provided in this disclosure.

[0090] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0091] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0092] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0093] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0094] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0098] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A data transmission method, characterized in that, Applied to head-mounted display devices, the method includes: Receive a data acquisition request sent by a target application; wherein the data acquisition request is used to acquire target data of the head-mounted display device, and the target data includes camera data and sensor data; In response to the data acquisition request, the target application is controlled to acquire the target data through the camera service of the head-mounted display device.

2. The method according to claim 1, characterized in that, In response to the data acquisition request, controlling the target application to acquire the target data through the camera service of the head-mounted display device includes: In response to the data acquisition request, the target application is controlled to acquire the camera data through the camera service, and the target application is controlled to acquire the sensor data through the sensor data callback interface of the camera service.

3. The method according to claim 2, characterized in that, The step of controlling the target application to acquire camera data through the camera service in response to the data acquisition request, and controlling the target application to acquire sensor data through the sensor data callback interface of the camera service, includes: In response to the data acquisition request, the target application is controlled to acquire the camera data through the camera service and the camera hardware abstraction layer of the head-mounted display device, and the target application is controlled to acquire the sensor data through the sensor data callback interface of the camera service and the low-latency interface of the camera hardware abstraction layer.

4. The method according to claim 3, characterized in that, The camera hardware abstraction layer acquires camera data through the image signal processor of the head-mounted display device, and the low-latency interface of the camera hardware abstraction layer acquires sensor data through the digital signal processor of the head-mounted display device.

5. The method according to claim 1, characterized in that, The method further includes: The target data read is encoded using the camera service to obtain the encoded target data; The encoded target data is transmitted to the target application via the camera service.

6. The method according to claim 1, characterized in that, After responding to the data acquisition request and controlling the target application to acquire the target data through the camera service of the head-mounted display device, the method further includes: Control the target application to perform the target operation based on the target data.

7. The method according to claim 6, characterized in that, The target operations include head tracking and gesture recognition.

8. A data transmission device, characterized in that, For use in head-mounted display devices, the device includes: A receiving module is used to receive a data acquisition request sent by a target application; wherein the data acquisition request is used to acquire target data of the head-mounted display device, and the target data includes camera data and sensor data; The acquisition module is used to respond to the data acquisition request and control the target application to acquire the target data through the camera service of the head-mounted display device.

9. A head-mounted display device, characterized in that, The head-mounted display device includes: Memory is used to store executable computer instructions; A processor configured to execute the data transmission method according to any one of claims 1-7, under the control of the executable computer instructions.

10. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, perform the data transmission method according to any one of claims 1-7.

Images