Camera data transmission method and apparatus, electronic device, and medium

Abstract

Embodiments of the present disclosure disclose a camera data transmission method and device, electronic equipment and medium, the method comprising: in response to a camera data acquisition instruction, detecting whether a target driver is in a running state; in the case that the target driver is in the running state and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras of a head-mounted display device respectively, reading the camera data corresponding to the N cameras respectively from a camera hardware abstraction layer of the head-mounted display device through a camera service of the head-mounted display device, performing a merging process on the camera data corresponding to the N cameras respectively to obtain merged camera data, and sending the merged camera data to a terminal device; in the case that the target driver is in the running state and the camera data acquisition instruction is used to acquire camera data of one of the cameras, reading the camera data corresponding to the one camera from the camera hardware abstraction layer through the camera service, and sending the camera data to the terminal device.

Description

Technical Field

[0001] This disclosure relates to the field of electronic device technology, and more specifically, to a camera data transmission method, a camera data transmission device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] To reduce the power consumption of AR products such as AR glasses, current methods typically connect AR glasses to mobile devices. When an application is running on the mobile device, the device projects the application's content to the AR glasses for display. However, mobile devices usually can only use data from their own camera and cannot select to use data from different cameras on the AR glasses based on user needs. Summary of the Invention

[0003] The purpose of this disclosure is to provide a camera data transmission method, apparatus, electronic device, and medium.

[0004] According to a first aspect of the present disclosure, a camera data transmission method is provided, the method comprising:

[0005] In response to a camera data acquisition command sent by a terminal device, the system detects whether the target driver is running; wherein the target driver is used to virtualize the head-mounted display device as a camera device.

[0006] When the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, the camera data corresponding to the N cameras is read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and the camera data corresponding to the N cameras is merged to obtain merged camera data and sent to the terminal device; N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device;

[0007] When the target driver is running and the camera data acquisition instruction is used to acquire camera data of one of the M cameras, the camera service reads the camera data corresponding to one of the cameras from the camera hardware abstraction layer and sends it to the terminal device.

[0008] Optionally, the step of reading camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and merging the camera data corresponding to the N cameras to obtain merged camera data and sending it to the terminal device includes:

[0009] The camera data corresponding to each of the N cameras is read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device;

[0010] The target merging method transmitted by the terminal device is obtained through the camera service, and the camera data corresponding to the N cameras are merged based on the target merging method to obtain merged camera data and send it to the terminal device.

[0011] The target merging method is either an up-and-down superposition method or a left-and-right superposition method.

[0012] According to a second aspect of the present disclosure, a camera data transmission method is provided, the method comprising:

[0013] The device receives camera data from a head-mounted display device; wherein the camera data is camera data from one of M cameras acquired by the head-mounted display device based on a camera data acquisition instruction sent by a terminal device; or, the camera data is merged camera data obtained by the head-mounted display device by merging the camera data corresponding to N cameras from the M cameras acquired based on a camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras in the head-mounted display device.

[0014] When the first camera hardware abstraction layer of the terminal device is in working state, the camera data is sent to the camera service of the terminal device through the first camera hardware abstraction layer;

[0015] The camera service controls the sending of camera data to the target application via the camera interface of the terminal device.

[0016] Optionally, the method further includes:

[0017] Receive the user's first trigger action;

[0018] In response to the first triggering operation, the first camera hardware abstraction layer of the terminal device is activated, and a camera data acquisition instruction is sent to the head-mounted display device;

[0019] Wherein, the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras; or,

[0020] The camera data acquisition instruction is used to acquire camera data from one of the M cameras.

[0021] Optionally, when the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras, the method further includes:

[0022] In response to the first triggering operation, a target merging mode is sent to the head-mounted display device, so that the head-mounted display device can perform merging processing on the camera data corresponding to the N cameras respectively based on the target merging mode to obtain merged camera data;

[0023] The target merging method is either vertical or horizontal.

[0024] Optionally, the method further includes:

[0025] Receive the user's second trigger action;

[0026] In response to the second triggering operation, the second camera hardware abstraction layer of the terminal device is activated;

[0027] When the second camera hardware abstraction layer is in working state, the camera data of the terminal device is obtained through the second camera hardware abstraction layer, and the camera data of the terminal device is sent to the camera service;

[0028] The camera service controls the sending of camera data from the terminal device to the target application via the camera interface.

[0029] According to a third aspect of the present disclosure, a camera data transmission apparatus is provided, the apparatus comprising:

[0030] The detection module is used to detect whether the target driver is running in response to the camera data acquisition command sent by the terminal device; wherein, the target driver is used to virtualize the head-mounted display device as a camera device;

[0031] The first processing module is configured to, when the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, merge the camera data corresponding to the N cameras, and send the merged camera data to the terminal device; N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device;

[0032] The second processing module is used to read the camera data corresponding to one of the cameras from the camera hardware abstraction layer through the camera service and send it to the terminal device when the target driver is running and the camera data acquisition instruction is used to acquire camera data of one of the M cameras.

[0033] According to a fourth aspect of the present disclosure, a camera data transmission apparatus is provided, the apparatus comprising:

[0034] A receiving module is configured to receive camera data from a head-mounted display device sent by the head-mounted display device; wherein the camera data is camera data from one of M cameras acquired by the head-mounted display device based on a camera data acquisition instruction sent by a terminal device; or, the camera data is merged camera data obtained by the head-mounted display device by merging the camera data corresponding to N cameras from the M cameras acquired based on a camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras in the head-mounted display device;

[0035] The sending module is used to send the camera data to the camera service of the terminal device through the first camera hardware abstraction layer when the first camera hardware abstraction layer of the terminal device is in working state.

[0036] The control module is used to control the camera service to send the camera data to the target application through the camera interface of the terminal device.

[0037] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0038] Memory is used to store executable computer instructions;

[0039] A processor, configured to execute the camera data transmission method according to the first or second aspect above, under the control of the executable computer instructions.

[0040] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processor, perform the camera data transmission method described in the first or second aspect above.

[0041] One beneficial effect of this disclosure is that, when a head-mounted display device receives a camera data acquisition instruction from a terminal device, if the target driver is running and the instruction is for acquiring camera data corresponding to N cameras out of M cameras on the head-mounted display device, it will read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the head-mounted display device's camera service, and merge the data to obtain merged camera data, which will then be sent to the terminal device. Here, N is an integer greater than 1 and less than M, and M is the total number of cameras on the head-mounted display device. If the target driver is running and the instruction is for acquiring camera data from one of the M cameras, it will read the corresponding camera data from the camera hardware abstraction layer of the head-mounted display device through the head-mounted display device's camera service and send it to the terminal device. In this way, the terminal device can acquire camera data from different cameras on the head-mounted display device according to different user needs.

[0042] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0044] Figure 1 This is a schematic diagram of the hardware configuration of a camera data transmission system according to an embodiment of the present disclosure;

[0045] Figure 2 This is a schematic flowchart of a camera data transmission method according to an embodiment of the present disclosure;

[0046] Figure 3 This is a schematic diagram illustrating an application scenario of the camera data transmission method according to an embodiment of the present disclosure;

[0047] Figure 4a This is a schematic diagram of a target merging method according to an example of this disclosure;

[0048] Figure 4b This is a schematic diagram of a target merging method according to another example of this disclosure;

[0049] Figure 4c This is a schematic diagram of merging camera data according to an example of this disclosure;

[0050] Figure 4d This is a schematic diagram of merging camera data according to another example of this disclosure;

[0051] Figure 5 This is a schematic flowchart of a camera data transmission method according to another embodiment of the present disclosure;

[0052] Figure 6 This is a schematic block diagram of a camera data transmission device according to an embodiment of the present disclosure;

[0053] Figure 7 This is a schematic block diagram of a camera data transmission device according to another embodiment of the present disclosure;

[0054] Figure 8 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0055] Figure 1 This is a schematic diagram of the hardware configuration that can be used to implement a camera data transmission system in one embodiment. Figure 1 A head-mounted display device 100, a terminal device 200, and a network 300 are illustrated. The head-mounted display device 100 can connect to the network 300 and can also connect to the terminal device 200 via a communication method such as Bluetooth. In one embodiment, the head-mounted display device 100 connects to the terminal device 200 only via a communication method such as Bluetooth. Multiple servers 301 and 302 can be configured in the network 300. The network 300 can be a wireless communication network or a wired communication network. The network 300 can be a local area network (LAN) or a wide area network (WAN). The network 300 can be for short-range communication or long-range communication.

[0056] In one embodiment, such as Figure 1 As shown, the head-mounted display device 100 may include a processor 101 and a memory 102. The head-mounted display device 100 also includes a communication device 103, a display device 104, a user interface 105, a camera device 106, an audio / video interface 107, and sensors 108, etc. Furthermore, the head-mounted display device 100 may also include a power management chip 109 and a battery 110, etc.

[0057] The processor 101 can be any type of processor. The memory 102 can store the underlying software, system software, application software, data, etc., required for the operation of the head-mounted display device 100. The memory 102 can include various forms of memory, such as ROM, RAM, Flash, etc. The communication device 103 can include, for example, a WiFi communication device, a Bluetooth communication device, a 3G, 4G, or 5G communication device. Through the communication device 103, the head-mounted display device 100 can be deployed in a network. The display device 104 can be a liquid crystal display (LCD), an OLED display, etc. In one example, the display device 104 can be a touchscreen. Users can perform input operations through the display device 104. Furthermore, users can also perform fingerprint recognition, etc., through the touchscreen. The user interface 105 can include a USB interface, a Lightning interface, a keyboard, etc. The camera device 106 can be a single camera or multiple cameras. The audio / video interface 107 can include, for example, a speaker interface, a microphone interface, a video transmission interface such as HDMI, etc. The sensor 108 can include, for example, a gyroscope, an accelerometer, a temperature sensor, a humidity sensor, a pressure sensor, etc. For example, the sensor can determine the posture information of the head-mounted display device. The power management chip 109 can be used to manage the power input to the head-mounted display device 100, and can also manage the battery 110 to ensure maximum utilization efficiency. The battery 110 is, for example, a lithium-ion battery.

[0058] The head-mounted display device 100 can be AR (Augmented Reality) glasses or MR (Mixed Reality) glasses, etc. The AR glasses can be split AR glasses, and the MR glasses can be split MR glasses. This disclosure does not limit the specific type of glasses. Figure 1 The components shown are merely illustrative. The head-mounted display device 100 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The head-mounted display device 100 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0059] In this embodiment, the memory 102 of the head-mounted display device 100 is used to store program instructions that control the processor 101 to perform the camera data transmission method. Those skilled in the art can design these instructions based on the disclosed solution. How the instructions control the processor is well-known in the art and will not be described in detail here.

[0060] In one embodiment, such as Figure 1As shown, the terminal device 200 may include a processor 201 and a memory 202. The terminal device 200 also includes a communication device 203, a display device 204, a user interface 205, a camera device 206, an audio / video interface 207, and a sensor 208, etc. Furthermore, the terminal device 200 may also include a power management chip 209 and a battery 210, etc.

[0061] The terminal device 200 can be a mobile phone, laptop, tablet computer, PDA, wearable device, etc., and this embodiment does not limit the scope of the application. For example, the terminal device can be an Android device, such as an Android device with a display device 204, or an Android device without a display device 204. It should be noted that if the terminal device does not have a display device 201, the application content of the application running on the terminal device can be displayed through a head-mounted display device 100. That is, Figure 1 The components shown are merely illustrative. Terminal device 200 may include... Figure 1 One or more of the components shown, but not necessarily including Figure 1 All components in it. Figure 1 The terminal device 200 shown is merely illustrative and is by no means intended to limit the embodiments, applications, or uses herein.

[0062] In this embodiment, the memory 202 of the terminal device 200 is used to store program instructions that control the processor 201 to perform the camera data transmission method. Those skilled in the art can design these instructions based on the disclosed solution. How the instructions control the processor is well-known in the art and will not be described in detail here.

[0063] It should be understood that, despite Figure 1 Only one head-mounted display device 100 and one terminal device 200 are shown, but this does not mean that the number of each is limited. The camera data transmission system may contain multiple head-mounted display devices 100 and multiple terminal devices 200.

[0064] In the above description, those skilled in the art can design instructions based on the solutions provided in this disclosure. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0065] <Method Example 1>

[0066] Figure 2 This disclosure illustrates a camera data transmission method according to an embodiment of the present disclosure. This camera data transmission method can be... Figure 1 The illustrated head-mounted display device can be AR glasses. Figure 2As shown, the camera data transmission method of this embodiment may include the following steps S2100 to S2300:

[0067] Step S2100: In response to the camera data acquisition command sent by the terminal device, detect whether the target driver is running.

[0068] In this embodiment, before transmitting camera data to the terminal device, the head-mounted display device needs to establish a communication connection with the terminal device. This communication connection can be a Universal Serial Bus (USB) connection. The head-mounted display device can then transmit USB Video Class (UVC) data to the terminal device via the USB connection, whereby the UVC data includes camera data.

[0069] Among them, the camera data acquisition instruction is the instruction sent by the terminal device to the head-mounted display device based on the user's first trigger operation.

[0070] Optionally, the camera data acquisition instruction can be used to acquire camera data corresponding to N cameras out of M cameras in the head-mounted display device, where N is an integer greater than 1 and less than M, and M is the total number of cameras in the head-mounted display device. Specifically, after receiving the user's first trigger operation, the terminal device can respond to the first trigger operation by sending a camera data acquisition instruction to the head-mounted display device to acquire camera data corresponding to the N cameras of the head-mounted display device. The head-mounted display device can then acquire the camera data corresponding to the N cameras based on the camera data acquisition instruction.

[0071] Example 1: After receiving a user's trigger operation targeting both camera ID1 and camera ID2, the terminal device can respond to the trigger operation by sending a camera data acquisition command to the head-mounted display device to simultaneously acquire camera data from both camera ID1 and camera ID2, since camera ID0 corresponds to camera 1 of the head-mounted display device and camera ID2 corresponds to camera 2 of the head-mounted display device.

[0072] Optionally, the camera data acquisition command can also be used to acquire camera data from one of the M cameras in the head-mounted display device. Specifically, after receiving a first trigger operation from the user, the terminal device can respond to the first trigger operation by sending a camera data acquisition command to the head-mounted display device to acquire camera data from one of the cameras in the head-mounted display device. The head-mounted display device can then acquire the camera data from that camera based on the camera data acquisition command.

[0073] Example 2: After receiving a user's trigger operation for camera ID1, since camera ID0 corresponds to camera 1 of the head-mounted display device, the terminal device can respond to the trigger operation by sending a camera data acquisition command to the head-mounted display device to acquire camera data of camera 1 of the head-mounted display device.

[0074] The target driver is used to virtualize the head-mounted display device as a camera device. This target driver can be a gadget program or a gadget service.

[0075] Specifically, refer to Figure 3 After receiving a camera data acquisition command from a terminal device, the head-mounted display device can respond by checking whether the gadget program is running. If the gadget program is not running, the head-mounted display device can start it. If the gadget program is running, the head-mounted display device can then function as a virtual USB camera device.

[0076] It should be noted that when the head-mounted display is virtualized as a USB camera device, the terminal device can see the USB camera device node, but there is no camera data in this node.

[0077] After executing the S2100 response to the camera data acquisition command sent by the terminal device and detecting whether the target driver is running, it proceeds to:

[0078] Step S2200: When the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, the camera data corresponding to the N cameras is read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and the camera data corresponding to the N cameras is merged to obtain merged camera data and sent to the terminal device.

[0079] In this embodiment, since USB UVC transmission can usually only transmit camera data from one camera, if the terminal device needs to acquire camera data from N cameras simultaneously, then when the target driver is running, the camera data corresponding to each of the N cameras is first read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and then the camera data corresponding to each of the N cameras is merged to obtain merged camera data.

[0080] In an optional embodiment, step S2200, which involves reading camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device via the camera service of the head-mounted display device and merging the camera data corresponding to the N cameras to obtain merged camera data and sending it to the terminal device, may further include: when the target driver is running, reading camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device via the camera service of the head-mounted display device; obtaining the target merging method transmitted by the terminal device via the camera service; and merging the camera data corresponding to the N cameras based on the target merging method to obtain merged camera data and sending it to the terminal device.

[0081] The target merging method is either a vertical stacking method or a horizontal stacking method. (Refer to...) Figure 4a When the target merging method is a top-to-bottom overlay method, the camera service of the head-mounted display device will merge the camera data corresponding to Camera 1, Camera 2...Camera N using a top-to-bottom overlay method. (Refer to...) Figure 4b When the target merging method is left-right overlay, the camera service of the head-mounted display device will merge the camera data corresponding to camera 1, camera 2, ..., camera N using the left-right overlay method.

[0082] Continuing with Example 1 above, refer to... Figure 3 When the gadget program is running, the head-mounted display device also needs to check whether its camera service is running. If the camera service is not running, the head-mounted display device can then enable the camera service. When the camera service is running, it can read camera data from camera 1 and camera 2 from the camera hardware abstraction layer (Camera HAL). (And refer to...) Figure 4c The camera data from camera 1 and camera 2 are combined using a superposition method to obtain combined camera data 41. Alternatively, refer to... Figure 4d The camera data from camera 1 and camera 2 are combined using a left-right overlay method to obtain combined camera data 42.

[0083] It should be noted that when the head-mounted display device sends camera data to the terminal device through the first interface, the camera data can be written to the USB camera device node, where the camera data is stored.

[0084] Continuing with Example 1 above, refer to... Figure 3 Camera service for head-mounted displays will Figure 4c The data from the merging camera shown is 41 or Figure 4d The combined camera data 42 shown is sent to the first buffer corresponding to the gadget program for storage. The USB driver of the head-mounted display device sends the combined camera data 41 or combined camera data 42 stored in the first buffer to the USB driver of the terminal device.

[0085] Continuing with Example 1 above, refer to... Figure 3 The terminal device's USB driver receives either merged camera data 41 or merged camera data 42, and sends either merged camera data 41 or merged camera data 42 to the terminal device's camera service through the enabled first camera hardware abstraction layer (Camera HAL). The terminal device's camera service then sends either merged camera data 41 or merged camera data 42 to the target application through the camera application program interface (Camera API).

[0086] Alternatively, the terminal device's camera service may first split the merged camera data 41 or merged camera data 42 to obtain camera data from camera 1 and camera data from camera 2, and then send the camera data from camera 1 and camera data from camera 2 to the target application through the camera API.

[0087] It should be noted that the target application can be the same application or different applications; this embodiment does not limit this.

[0088] Step S2300: When the target driver is running and the camera data acquisition instruction is used to acquire camera data of one of the M cameras, the camera service reads the camera data corresponding to one of the cameras from the camera hardware abstraction layer and sends it to the terminal device.

[0089] Continuing with Example 2 above, refer to... Figure 3While the gadget program is running, the head-mounted display device also needs to check whether its camera service is running. If the camera service is not running, the head-mounted display device can then enable the camera service. When the camera service is running, it can read camera data from camera 1 from the camera hardware abstraction layer (Camera HAL).

[0090] Continuing with Example 2 above, refer to... Figure 3 The camera service of the head-mounted display sends the camera data from camera 1 to the first buffer corresponding to the gadget program for storage. The USB driver of the head-mounted display then sends the camera data from camera 1 stored in the first buffer to the USB driver of the terminal device.

[0091] Continuing with Example 2 above, refer to... Figure 3 The terminal device's USB driver receives camera data from camera 1 and sends the camera data from camera 1 to the terminal device's camera service through the enabled first camera hardware abstraction layer (Camera HAL). The terminal device's camera service then sends the camera data from camera 1 to the target application via the camera API.

[0092] According to embodiments of this disclosure, when a head-mounted display device receives a camera data acquisition instruction from a terminal device, if the target driver is running and the instruction is for acquiring camera data corresponding to N cameras out of M cameras on the head-mounted display device, the device will read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the device's camera service. The device will then merge the N cameras' data to obtain merged camera data, which will be sent to the terminal device. Here, N is an integer greater than 1 and less than M, and M is the total number of cameras on the head-mounted display device. If the target driver is running and the instruction is for acquiring camera data from one of the M cameras, the device will read the corresponding camera data from the camera hardware abstraction layer of the head-mounted display device through the device's camera service and send it to the terminal device. In this way, the terminal device can acquire camera data from different cameras on the head-mounted display device according to different user needs.

[0093] <Method Example 2>

[0094] Figure 5 This disclosure illustrates a camera data transmission method according to an embodiment of the present disclosure. This camera data transmission method can be... Figure 1 The terminal equipment shown is implemented as follows. Figure 5 As shown, the camera data transmission method of this embodiment may include the following steps S5100 to S5300:

[0095] Step S5100: Receive camera data from the head-mounted display device.

[0096] In this embodiment, a communication connection is established between the terminal device and the head-mounted display device, and the communication connection between the terminal device and the head-mounted display device can be a USB connection.

[0097] The camera data of the head-mounted display device is obtained by the head-mounted display device based on the camera data acquisition command sent by the terminal device.

[0098] Optionally, the camera data may be camera data from one of the M cameras acquired by the head-mounted display device based on a camera data acquisition command sent by the terminal device.

[0099] Optionally, the camera data can also be merged camera data obtained by the head-mounted display device merging the camera data corresponding to N cameras out of the M cameras based on the camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device.

[0100] In this embodiment, when the terminal device needs to use camera data from the head-mounted display device, the terminal device receives a first trigger operation from the user. On one hand, in response to this first trigger operation, the terminal device activates its first camera hardware abstraction layer. On the other hand, in response to this first trigger operation, the terminal device sends a camera data acquisition command to the head-mounted display device.

[0101] Wherein, the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras; or, the camera data acquisition instruction is used to acquire camera data of one of the M cameras. Furthermore, when the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras, the method further includes: in response to the first triggering operation, sending a target merging mode to the head-mounted display device, so that the head-mounted display device can perform merging processing on the camera data corresponding to the N cameras based on the target merging mode to obtain merged camera data.

[0102] The first camera hardware abstraction layer is used to send camera data from the head-mounted display device to the camera service of the terminal device. It should be noted that after the terminal device receives the user's first trigger operation, it can run an external camera provider to enable the terminal device's first camera hardware abstraction layer (Camera HAL).

[0103] Example 1, see reference Figure 3 After receiving a camera data acquisition command from the terminal device specifying the target merging method and the acquisition of camera data from cameras 1 and 2, the head-mounted display device will respond to the command by checking whether the gadget program is running. If the gadget program is not running, the head-mounted display device can start the gadget program. If the gadget program is running, the head-mounted display device can then function as a virtual USB camera device.

[0104] Reference Figure 3When the gadget program is running, the head-mounted display device also needs to check if its camera service is running. If the camera service is not running, the head-mounted display device can then enable the camera service. When the camera service is running, it can read camera data from camera 1 and camera 2 from the camera hardware abstraction layer (Camera HAL). (And refer to...) Figure 4c The camera data from camera 1 and camera 2 are combined using a superposition method to obtain combined camera data 41. Alternatively, refer to... Figure 4d The camera data from camera 1 and camera 2 are combined using a left-right overlay method to obtain combined camera data 42.

[0105] Reference Figure 3 Camera service for head-mounted displays will Figure 4c The data from the merging camera shown is 41 or Figure 4d The combined camera data 42 shown is sent to the first buffer corresponding to the gadget program for storage. The USB driver of the head-mounted display device sends the combined camera data 41 or combined camera data 42 stored in the first buffer to the USB driver of the terminal device.

[0106] Example 2, see reference Figure 3 After receiving a camera data acquisition command from the terminal device, the head-mounted display device will respond to the command by checking whether the gadget program is running. If the gadget program is not running, the head-mounted display device can start the gadget program. If the gadget program is running, the head-mounted display device can then function as a virtual USB camera device.

[0107] Reference Figure 3While the gadget program is running, the head-mounted display device also needs to check whether its camera service is running. If the camera service is not running, the head-mounted display device can then enable the camera service. When the camera service is running, it can read camera data from camera 1 from the camera hardware abstraction layer (Camera HAL).

[0108] Reference Figure 3 The camera service of the head-mounted display sends the camera data from camera 1 to the first buffer corresponding to the gadget program for storage. The USB driver of the head-mounted display then sends the camera data from camera 1 stored in the first buffer to the USB driver of the terminal device.

[0109] Step S5200: When the first camera hardware abstraction layer of the terminal device is in working state, the camera data is sent to the camera service of the terminal device through the first camera hardware abstraction layer.

[0110] Continuing with Example 1 above, refer to... Figure 3 When the first camera hardware abstraction layer (Camera HAL) of the terminal device is in working condition, it can obtain... Figure 4c The data from the merging camera shown is 41 or Figure 4d The merged camera data 42 shown is sent to the camera service of the terminal device, and the merged camera data 41 or the merged camera data 42 is sent to the camera service of the terminal device.

[0111] Continuing with Example 2 above, refer to... Figure 3 When the first camera hardware abstraction layer (Camera HAL) of the terminal device is in working state, the camera data of camera 1 can be obtained through the first camera hardware abstraction layer (Camera HAL) and the camera data of camera 1 can be sent to the camera service of the terminal device.

[0112] Step S5300: Control the camera service to send the camera data to the target application through the camera interface of the terminal device.

[0113] The target application is any application currently running on the terminal device, including but not limited to games and video applications.

[0114] Continuing with Example 1 above, refer to... Figure 3 The terminal device's camera service receives merged camera data 41 or merged camera data 42 and sends the merged camera data 41 or merged camera data 42 to the target application through the camera API.

[0115] Alternatively, the terminal device's camera service may first split the merged camera data 41 or merged camera data 42 to obtain camera data from camera 1 and camera data from camera 2, and then send the camera data from camera 1 and camera data from camera 2 to the target application through the camera API.

[0116] Continuing with Example 2 above, refer to... Figure 3 The terminal device's camera service receives camera data from camera 1 and sends the camera data from camera 1 to the target application through the camera API.

[0117] According to embodiments of this disclosure, a terminal device can select to obtain camera data from one of the M cameras of a head-mounted display device, or obtain camera data corresponding to N cameras among the M cameras of the head-mounted display device, based on user needs. After receiving the camera data from the head-mounted display device, the terminal device can send the camera data from the head-mounted display device to the camera service of the terminal device through the enabled first camera hardware abstraction layer, and control the camera service to send the camera data from the head-mounted display device to the target application through the camera interface of the terminal device.

[0118] In one embodiment, the camera data transmission method of this disclosure further includes the following steps S6100 to S6400:

[0119] Step S6100: Receive the user's second trigger operation.

[0120] Specifically, when the terminal device needs to use its own camera data, the terminal device will receive a second trigger operation from the user.

[0121] Step S6200: In response to the second triggering operation, the second camera hardware abstraction layer of the terminal device is activated.

[0122] The second camera hardware abstraction layer is used to send the terminal device's own camera data to the terminal device's camera service. Specifically, after receiving the user's second trigger operation, the terminal device can respond to the second trigger operation by running the internal Camera Provider, thereby enabling the terminal device's second camera hardware abstraction layer (Camera HAL) through the internal Camera Provider.

[0123] Step S6300: When the second camera hardware abstraction layer is in working state, the camera data of the terminal device is obtained through the second camera hardware abstraction layer, and the camera data of the terminal device is sent to the camera service.

[0124] Specifically, when the second camera hardware abstraction layer (Camera HAL) of the terminal device is in working condition, the camera data of the terminal device can be obtained through the second camera hardware abstraction layer (Camera HAL) and sent to the camera service of the terminal device.

[0125] Step S6400: Send the camera data of the terminal device to the target application through the camera service of the terminal device.

[0126] Specifically, the terminal device's camera service receives camera data from the terminal device and sends the camera data to the target application through the terminal device's camera API, so that the target application can use the camera data from the head-mounted display device.

[0127] According to this embodiment, the terminal device can obtain its own camera data and the camera data of the head-mounted display device, and the two use different Camera Providers, which can avoid mutual interference between them.

[0128] <Device Example 1>

[0129] Figure 6 This is a schematic diagram of a camera data transmission device according to one embodiment, with reference to... Figure 6 As shown, the camera data transmission device 600 includes a detection module 610, a first processing module 620, and a second processing module 630.

[0130] The detection module 610 is used to detect whether the target driver is running in response to a camera data acquisition command sent by the terminal device; wherein, the target driver is used to virtualize the head-mounted display device as a camera device;

[0131] The first processing module 620 is configured to, when the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, merge the camera data corresponding to the N cameras, and send the merged camera data to the terminal device; N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device;

[0132] The second processing module 630 is used to read the camera data corresponding to one of the cameras from the camera hardware abstraction layer through the camera service and send it to the terminal device when the target driver is running and the camera data acquisition instruction is used to acquire the camera data of one of the M cameras.

[0133] In one embodiment, the first processing module 620 is specifically configured to read camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device; obtain the target merging method transmitted by the terminal device through the camera service; and perform merging processing on the camera data corresponding to the N cameras based on the target merging method to obtain merged camera data and send it to the terminal device.

[0134] The target merging method is either an up-and-down superposition method or a left-and-right superposition method.

[0135] According to embodiments of this disclosure, when a head-mounted display device receives a camera data acquisition instruction from a terminal device, if the target driver is running and the instruction is for acquiring camera data corresponding to N cameras out of M cameras on the head-mounted display device, the device will read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the device's camera service. The device will then merge the N cameras' data to obtain merged camera data, which will be sent to the terminal device. Here, N is an integer greater than 1 and less than M, and M is the total number of cameras on the head-mounted display device. If the target driver is running and the instruction is for acquiring camera data from one of the M cameras, the device will read the corresponding camera data from the camera hardware abstraction layer of the head-mounted display device through the device's camera service and send it to the terminal device. In this way, the terminal device can acquire camera data from different cameras on the head-mounted display device according to different user needs.

[0136] <Device Embodiment Two>

[0137] Figure 7 This is a schematic diagram of a camera data transmission device according to one embodiment, with reference to... Figure 7 As shown, the camera data transmission device 700 includes a receiving module 710, a sending module 720, and a control module 730.

[0138] The receiving module 710 is configured to receive camera data from the head-mounted display device sent by the head-mounted display device; wherein the camera data is camera data of one of the M cameras acquired by the head-mounted display device based on a camera data acquisition instruction sent by a terminal device; or, the camera data is merged camera data obtained by the head-mounted display device by merging the camera data corresponding to N cameras among the M cameras acquired based on a camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device;

[0139] The sending module 720 is used to send the camera data to the camera service of the terminal device through the first camera hardware abstraction layer when the first camera hardware abstraction layer of the terminal device is in working state.

[0140] The control module 730 is used to control the camera service to send the camera data to the target application through the camera interface of the terminal device.

[0141] In one embodiment, the receiving module 710 is further configured to receive the user's first trigger operation;

[0142] The sending module 720 is configured to respond to the first triggering operation, activate the first camera hardware abstraction layer of the terminal device, and send the camera data acquisition instruction to the head-mounted display device;

[0143] Wherein, the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras; or,

[0144] The camera data acquisition instruction is used to acquire camera data from one of the M cameras.

[0145] In one embodiment, when the camera data acquisition instruction is used to acquire camera data corresponding to N cameras among the M cameras, the sending module 720 is further configured to respond to the first triggering operation by sending a target merging method to the head-mounted display device, so that the head-mounted display device can perform merging processing on the acquired camera data corresponding to the N cameras based on the target merging method to obtain merged camera data.

[0146] The target merging method is either vertical or horizontal.

[0147] In one embodiment, the receiving module 710 is further configured to receive a second trigger operation from the user;

[0148] The control module 730 is also configured to activate the second camera hardware abstraction layer of the terminal device in response to the second triggering operation;

[0149] The sending module 720 is also configured to, when the second camera hardware abstraction layer is in working state, obtain camera data of the terminal device through the second camera hardware abstraction layer, and send the camera data of the terminal device to the camera service;

[0150] The receiving module 730 is also configured to control the camera service to send camera data from the terminal device to the target application through the camera interface.

[0151] According to embodiments of this disclosure, a terminal device can select to obtain camera data from one of the M cameras of a head-mounted display device, or obtain camera data corresponding to N cameras among the M cameras of the head-mounted display device, based on user needs. After receiving the camera data from the head-mounted display device, the terminal device can send the camera data from the head-mounted display device to the camera service of the terminal device through the enabled first camera hardware abstraction layer, and control the camera service to send the camera data from the head-mounted display device to the target application through the camera interface of the terminal device.

[0152] <Equipment Example>

[0153] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to one embodiment. For example... Figure 8 As shown, the electronic device 800 includes a processor 810 and a memory 820.

[0154] The memory 820 can be used to store executable computer instructions.

[0155] The processor 810 can be used to execute the camera data transmission method according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0156] The electronic device 800 can be as follows: Figure 1 The head-mounted display device 100 shown can also be Figure 1 The terminal device 200 shown can also be a device with other hardware structures, which are not limited here.

[0157] In another embodiment, the electronic device 800 may include the camera data transmission device 600 or the camera data transmission device 700.

[0158] In one embodiment, each module of the camera data transmission device 600 or camera data transmission device 700 can be implemented by the processor 810 running computer instructions stored in the memory 820.

[0159] Computer-readable storage media

[0160] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the camera data transmission method provided in this disclosure.

[0161] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0162] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0163] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0164] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0165] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0166] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0169] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for transmitting data from a camera, characterized in that, The method includes: In response to a camera data acquisition command sent by a terminal device, the system detects whether the target driver is running; wherein the target driver is used to virtualize the head-mounted display device as a camera device. When the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, the camera data corresponding to the N cameras is read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and the camera data corresponding to the N cameras is merged to obtain merged camera data and sent to the terminal device; N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device; When the target driver is running and the camera data acquisition instruction is used to acquire camera data of one of the M cameras, the camera service reads the camera data corresponding to one of the cameras from the camera hardware abstraction layer and sends it to the terminal device.

2. The method according to claim 1, characterized in that, The step of reading camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, and merging the camera data corresponding to the N cameras to obtain merged camera data and sending it to the terminal device includes: The camera data corresponding to each of the N cameras is read from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device; The target merging method transmitted by the terminal device is obtained through the camera service, and the camera data corresponding to the N cameras are merged based on the target merging method to obtain merged camera data and send it to the terminal device. The target merging method is either an up-and-down superposition method or a left-and-right superposition method.

3. A method for transmitting data from a camera, characterized in that, The method includes: The device receives camera data from a head-mounted display device; wherein the camera data is camera data from one of M cameras acquired by the head-mounted display device based on a camera data acquisition instruction sent by a terminal device; or, the camera data is merged camera data obtained by the head-mounted display device by merging the camera data corresponding to N cameras from the M cameras acquired based on a camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras in the head-mounted display device. When the first camera hardware abstraction layer of the terminal device is in working state, the camera data is sent to the camera service of the terminal device through the first camera hardware abstraction layer; The camera service controls the sending of camera data to the target application via the camera interface of the terminal device.

4. The method according to claim 3, characterized in that, The method further includes: Receive the user's first trigger action; In response to the first triggering operation, the first camera hardware abstraction layer of the terminal device is activated, and a camera data acquisition instruction is sent to the head-mounted display device; Wherein, the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras; or, The camera data acquisition instruction is used to acquire camera data from one of the M cameras.

5. The method according to claim 4, characterized in that, When the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of the M cameras, the method further includes: In response to the first triggering operation, a target merging mode is sent to the head-mounted display device, so that the head-mounted display device can perform merging processing on the camera data corresponding to the N cameras respectively based on the target merging mode to obtain merged camera data; The target merging method is either vertical or horizontal.

6. The method according to claim 3, characterized in that, The method further includes: Receive the user's second trigger action; In response to the second triggering operation, the second camera hardware abstraction layer of the terminal device is activated; When the second camera hardware abstraction layer is in working state, the camera data of the terminal device is obtained through the second camera hardware abstraction layer, and the camera data of the terminal device is sent to the camera service; The camera service controls the sending of camera data from the terminal device to the target application via the camera interface.

7. A camera data transmission device, characterized in that, The device includes: The detection module is used to detect whether the target driver is running in response to the camera data acquisition command sent by the terminal device; wherein, the target driver is used to virtualize the head-mounted display device as a camera device; The first processing module is configured to, when the target driver is running and the camera data acquisition instruction is used to acquire camera data corresponding to N cameras out of M cameras of the head-mounted display device, read the camera data corresponding to the N cameras from the camera hardware abstraction layer of the head-mounted display device through the camera service of the head-mounted display device, merge the camera data corresponding to the N cameras, and send the merged camera data to the terminal device; N is an integer greater than 1 and less than M, and M is the total number of cameras of the head-mounted display device; The second processing module is used to read the camera data corresponding to one of the cameras from the camera hardware abstraction layer through the camera service and send it to the terminal device when the target driver is running and the camera data acquisition instruction is used to acquire camera data of one of the M cameras.

8. A camera data transmission device, characterized in that, The device includes: A receiving module is configured to receive camera data from a head-mounted display device sent by the head-mounted display device; wherein the camera data is camera data from one of M cameras acquired by the head-mounted display device based on a camera data acquisition instruction sent by a terminal device; or, the camera data is merged camera data obtained by the head-mounted display device by merging the camera data corresponding to N cameras from the M cameras acquired based on a camera data acquisition instruction sent by the terminal device, where N is an integer greater than 1 and less than M, and M is the total number of cameras in the head-mounted display device; The sending module is used to send the camera data to the camera service of the terminal device through the first camera hardware abstraction layer when the first camera hardware abstraction layer of the terminal device is in working state. The control module is used to control the camera service to send the camera data to the target application through the camera interface of the terminal device.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable computer instructions; A processor, configured to execute the camera data transmission method according to any one of claims 1-6, under the control of the executable computer instructions.

10. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, perform the camera data transmission method according to any one of claims 1-6.

Images