Distributed device capability virtualization method, medium, and electronic device

Abstract

This application relates to the field of Internet of Things (IoT) technology, and particularly to a method, medium, and electronic device for distributed device capability virtualization. The method includes: establishing a first-level communication relationship between a first electronic device and a second electronic device, wherein the first-level communication relationship instructs the first electronic device to provide preset capabilities to the second electronic device; a third electronic device sending a request to the second electronic device to establish a second-level communication relationship; wherein the second-level communication relationship request is used by the third electronic device to obtain authorization from the second electronic device to utilize the preset capabilities through the second electronic device; and, if the second electronic device grants the third electronic device authorization to establish a second-level communication relationship with the second electronic device based on the second-level communication relationship request, a second-level communication relationship is established between the third electronic device and the second electronic device, wherein the second-level communication relationship instructs the third electronic device to utilize the preset capabilities through the second electronic device. This achieves multi-level sharing of device capabilities and improves the user experience.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and in particular to methods, media, and electronic devices for distributed device capability virtualization. Background Technology

[0002] Distributed device capability virtualization refers to the sharing of device capabilities (such as shooting, sound pickup, and sound playback) between two different electronic devices by treating components (e.g., camera, microphone, speaker, etc.) of one electronic device as virtual components in the other. This allows electronic devices in a distributed environment to share these capabilities. For example, a mobile phone can use the camera of a smart TV for video calls; or a mobile phone can use the screen of a tablet to display content while simultaneously using the tablet's camera for video calls.

[0003] Current technology only supports virtualization of device capabilities between two electronic devices. For example, a mobile phone uses the camera capabilities of a drone to shoot wide-angle video. However, in practical applications, there may be a need for two electronic devices to use the same device capabilities of the same electronic device. For example, multiple mobile phones may need to simultaneously use the camera capabilities of the same drone to shoot wide-angle video.

[0004] However, due to the structural and performance limitations of the electronic devices providing device capabilities, each device can only connect to one other electronic device. It cannot simultaneously connect to multiple electronic devices and transmit multiple identical data streams obtained through its device capabilities. Alternatively, if the distance between the device providing device capabilities and other electronic devices is too great, it cannot directly connect to other electronic devices and provide device capabilities via short-range communication technology, thus impacting the user experience. Summary of the Invention

[0005] This application provides a distributed device capability virtualization method, medium, and electronic device.

[0006] In a first aspect, embodiments of this application provide a distributed device capability virtualization method, the method comprising: establishing a first-level communication relationship between a first electronic device and a second electronic device, wherein the first-level communication relationship is used to instruct the first electronic device to provide a preset capability to the second electronic device;

[0007] The third electronic device sends a request to the second electronic device to establish a second-level communication relationship; wherein, the second-level communication relationship request is used by the third electronic device to obtain authorization from the second electronic device to utilize the preset capability through the second electronic device;

[0008] When the second electronic device requests permission from the third electronic device to establish a second-level communication relationship with the second electronic device based on the second-level communication relationship request, a second-level communication relationship is established between the third electronic device and the second electronic device, wherein the second-level communication relationship is used to instruct the third electronic device to utilize the preset capability through the second electronic device.

[0009] In this embodiment, the preset capabilities can be shooting capabilities, audio playback capabilities, audio acquisition capabilities, etc. The shooting capability will be used as an example below.

[0010] In this embodiment of the application, the first electronic device can be a drone, the second electronic device can be a mobile phone for the broadcast assistant, and the third electronic device can be a mobile phone for the broadcaster.

[0011] Once a first-level communication relationship is established between the drone and the anchor assistant's mobile phone, the drone can provide shooting capabilities to the anchor assistant's mobile phone, meaning the anchor assistant's mobile phone can control the drone's working status and data transmission status.

[0012] When a second-level communication relationship is established between the broadcaster's phone and the broadcaster's assistant's phone, the broadcaster's assistant's phone provides the broadcaster's phone with shooting capabilities. In other words, the broadcaster's phone can control the drone's working status and data transmission status through the broadcaster's assistant's phone.

[0013] For example, the anchor assistant's mobile phone can obtain video data corresponding to the footage captured by the drone. When the anchor assistant is watching the video, the anchor assistant can control the drone through the anchor assistant's mobile phone, such as adjusting the drone's shooting field of view and shooting angle, to select the best footage for the anchor.

[0014] The streamer can then access video data captured by the drone through the streamer assistant's phone, while simultaneously filming themselves using their own phone's front-facing camera. The phone will then merge the video data obtained from the streamer assistant's phone with the image data captured by their own phone's physical camera, enabling dual-view (simultaneous capture of two feeds, such as footage from the drone and footage from the streamer's phone) live streaming.

[0015] This solves the structural and performance limitations of the electronic devices providing device capabilities (e.g., drones). For example, an electronic device providing device capabilities (e.g., a drone) may only support connection to one electronic device and cannot simultaneously connect to multiple electronic devices (e.g., mobile phones) and transmit multiple identical data streams obtained through device capabilities. Furthermore, the distance between the electronic device providing device capabilities (e.g., drones) and other electronic devices may be too great, making it impossible to directly connect the electronic device providing device capabilities (e.g., drones) to other electronic devices (e.g., mobile phones) and provide device capabilities through short-range communication technologies. This enables multi-level sharing of device capabilities (e.g., mobile phones sharing video data), improving the user experience.

[0016] In another embodiment of this application, the first electronic device can be a mining truck, the second electronic device can be a computer that controls the movement of the mining truck to mine coal, and the third electronic device can be a mobile phone that monitors the movement video of the mining truck during the coal mining process via a computer.

[0017] Once a first-level communication relationship is established between the mining truck and the computer, the mining truck can provide the computer with shooting capabilities, meaning the computer can control the working status and / or data transmission status of the mining truck.

[0018] When a second-level communication relationship is established between a mobile phone and a computer, the computer provides the mobile phone with shooting capabilities, meaning that the mobile phone can control the working status of the mining truck and / or the data transmission status through the computer.

[0019] The computer can acquire video data of the minecart's path, allowing the user to control the minecart by manipulating its virtual camera, such as adjusting its direction and speed.

[0020] Users of mobile phones can then use a computer's virtual webcam to obtain video data captured by the mining truck and monitor the truck's operation during the coal mining process.

[0021] This solves the problem that, due to the structural and performance limitations of the electronic devices providing device capabilities (e.g., mining trucks), they can only connect to one electronic device at a time, and cannot simultaneously connect to multiple electronic devices (e.g., computers and mobile phones) and transmit multiple identical data streams obtained through device capabilities. Alternatively, it addresses the issue where the distance between the electronic device providing device capabilities (e.g., mining trucks) and other electronic devices (mobile phones) is too great to directly connect and provide device capabilities using short-range communication technology. This enables multi-level sharing of device capabilities (e.g., computers and mobile phones sharing video data), improving the user experience.

[0022] In one possible implementation of the above aspects, the method further includes: when the third electronic device sends a first invocation instruction to the second electronic device, and the second electronic device sends a second invocation instruction to the first electronic device according to the first invocation instruction;

[0023] The first electronic device provides the preset capability to the second electronic device according to the second invocation instruction;

[0024] The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction.

[0025] In this embodiment of the application, the first electronic device can be a drone, the second electronic device can be a mobile phone for the broadcast assistant, and the third electronic device can be a mobile phone for the broadcaster.

[0026] When the broadcaster's phone detects a user's operation that invokes the distributed capability virtualization, the broadcaster's phone sends a call command to the broadcaster's assistant's phone. The broadcaster's assistant's phone then sends a call command to the drone based on the broadcaster's command. The drone can then provide filming functionality to the broadcaster's assistant's phone based on these commands. In other words, the broadcaster's assistant's phone can control the drone's operating status and / or data transmission relationships. For example, the drone can send video data collected through filming to the broadcaster's assistant's phone. Alternatively, the broadcaster's assistant's phone can send control commands to the drone to adjust its field of view and shooting angle.

[0027] The assistant's phone provides drone shooting functions to the broadcaster's phone based on the commands sent from the broadcaster's phone. That is, the broadcaster's phone can control the drone's operating status and / or data transmission status through the assistant's phone. For example, the assistant's phone can send video data received from the drone to the broadcaster's phone. Alternatively, the broadcaster's phone can send control commands to the drone through the assistant's phone to adjust the drone's shooting field of view and shooting angle. Furthermore, the broadcaster's phone can send control commands to the assistant's phone to control the video data transmission status, such as pausing or disabling video data transmission.

[0028] In one possible implementation of the above aspects, the method further includes: the first electronic device providing the preset capability to the second electronic device according to the second invocation instruction;

[0029] The second electronic device provides a preset capability to the third electronic device according to the first invocation instruction, including: the first electronic device sending first data generated to the second electronic device to realize the preset capability;

[0030] The second electronic device sends the first data received from the first electronic device to the third electronic device.

[0031] In this embodiment, the preset capability can be a shooting capability, an audio acquisition capability, etc. If the preset capability is a shooting capability, the first data can be video data or image data; if the preset capability is an audio acquisition capability, the first data can be audio data. The following explanation uses the shooting capability as an example.

[0032] In a scenario where the broadcaster's phone sends a command to the broadcaster's assistant's phone, and the broadcaster's assistant's phone, based on the broadcaster's command, sends a command to the drone, the drone then provides filming capabilities to the broadcaster's phone according to the command from the broadcaster's assistant's phone. For example, the drone sends video data collected through filming to the broadcaster's assistant's phone. The broadcaster's assistant's phone then provides the drone's filming capabilities to the broadcaster's phone according to the command from the broadcaster's phone. For example, the broadcaster's assistant's phone sends video data collected by the drone using its filming capabilities to the broadcaster's phone.

[0033] In one possible implementation of the above aspects, the method further includes: the first electronic device providing the preset capability to the second electronic device according to the second invocation instruction;

[0034] The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction, including:

[0035] The third electronic device sends second data generated to the second electronic device to realize the preset capability;

[0036] The second electronic device sends the second data received from the third electronic device to the first electronic device.

[0037] In this embodiment, the first electronic device can be a speaker. The preset capability can be audio playback capability. The second data can be audio. The third electronic device sends audio data generated for audio playback to the second electronic device; the second electronic device sends the audio data received from the third electronic device to the speaker, and the speaker plays sound according to the audio data.

[0038] In one possible implementation of the above aspects, the method further includes: the preset capability includes one or more of the following: shooting capability, audio playback capability, and audio acquisition capability.

[0039] In one possible implementation of the above aspects, the method further includes: when the third electronic device obtains an instruction to establish a third-level communication relationship with the first electronic device, and the third electronic device sends an instruction to terminate the second-level communication relationship to the second electronic device according to the instruction to establish a third-level communication relationship, the second electronic device sends the device information of the first electronic device to the third electronic device according to the instruction to terminate the second-level communication relationship and closes the second-level communication relationship;

[0040] The third electronic device establishes a third-level communication relationship with the first electronic device based on the device information of the first electronic device, wherein the third-level communication relationship is used to instruct the first electronic device to provide a preset capability to the third electronic device.

[0041] Secondly, this application provides a distributed device capability virtualization method, which includes: a second electronic device receiving a request from a third electronic device to establish a second-level communication relationship;

[0042] When the second electronic device requests permission from the third electronic device to establish a second-level communication relationship with the second electronic device based on the second-level communication relationship request, the third electronic device establishes a second-level communication relationship with the second electronic device, wherein the second-level communication relationship is used to instruct the third electronic device to utilize the preset capabilities of the first electronic device through the second electronic device.

[0043] In one possible implementation of the above aspects, the method further includes: the second electronic device receiving a first invocation instruction sent by the third electronic device, and the second electronic device sending a second invocation instruction to the first electronic device according to the first invocation instruction;

[0044] The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction.

[0045] In one possible implementation of the above aspects, the second electronic device sends the first data received from the first electronic device to the third electronic device.

[0046] In one possible implementation of the above aspects, the second electronic device provides a preset capability to the third electronic device according to the first invocation instruction, including:

[0047] The second electronic device sends the second data received from the third electronic device to the first electronic device.

[0048] In one possible implementation of the above aspects, the preset capabilities include one or more of the following: shooting capability, audio playback capability, and audio acquisition capability.

[0049] In one possible implementation of the above aspects, the method further includes: when the third electronic device obtains an instruction to establish a third-level communication relationship with the first electronic device, and the third electronic device sends an instruction to terminate the second-level communication relationship to the second electronic device according to the instruction to establish a third-level communication relationship, the second electronic device sends the device information of the first electronic device to the third electronic device according to the instruction to terminate the second-level communication relationship and closes the second-level communication relationship.

[0050] Thirdly, embodiments of this application also provide a readable medium storing instructions that, when executed on an electronic device, implement either the first aspect or the second aspect of the distributed device capability virtualization method.

[0051] Fourthly, embodiments of this application also provide an electronic device, including:

[0052] Memory, used to store instructions; and

[0053] A processor, the processor being configured to execute the instructions to implement any possible implementation of the distributed device capability virtualization method of the second aspect. Attached Figure Description

[0054] Figure 1A According to some embodiments of this application, an application scenario diagram is shown where three electronic devices share video data through cascading.

[0055] Figure 1B According to some embodiments of this application, another application scenario diagram is shown where three electronic devices share video data through cascading.

[0056] Figure 2 According to some embodiments of this application, a system architecture diagram is shown for realizing multi-level wireless transmission of video data between mobile phone 300, mobile phone 200 and drone 100;

[0057] Figure 3 According to some embodiments of this application, a software architecture diagram is shown that enables a user (e.g., a broadcaster) of mobile phone 300 to obtain video data taken by drone 100 (e.g., a drone) through the virtual camera of mobile phone 200 and to take pictures of their own mobile phone 300 using the front camera of mobile phone 300.

[0058] Figure 4 According to some embodiments of this application, a schematic diagram of a process for establishing a cascading relationship between a drone 100, a mobile phone 200, and a mobile phone 300 is shown;

[0059] Figure 5According to some embodiments of this application, a schematic diagram of the control and data transmission authorization process of a camera in a drone 100 is shown;

[0060] Figure 6 According to some embodiments of this application, a schematic diagram of a cascading release process initiated by a drone 100 is shown;

[0061] Figure 7 According to some embodiments of this application, a schematic diagram of a cascading release process initiated by an intermediate device mobile phone 200 is shown;

[0062] Figure 8 According to some embodiments of this application, a structural schematic diagram of a drone 100 implementing the technical solution of this application is shown;

[0063] Figure 9 According to some embodiments of this application, a structural schematic diagram of a mobile phone 200 implementing the technical solution of this application is shown;

[0064] Figure 10 According to some embodiments of this application, a software structure block diagram of a mobile phone 200 is shown. Detailed Implementation

[0065] The illustrative embodiments of this application include, but are not limited to, distributed device capability virtualization methods, media, and electronic devices. The technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples.

[0066] To facilitate understanding, some terms used in the embodiments of this application will be introduced first.

[0067] Original equipment (Original Equipment): Equipment that provides the capabilities of an actual physical device. For example, the actual physical device can be a camera, microphone, speaker, etc., and the corresponding capabilities of the actual physical device can be shooting capability, sound pickup capability (audio acquisition capability), sound playback capability (audio playback capability, electroacoustic conversion capability, sound amplification capability), etc., but are not limited to these.

[0068] Level 1 devices: These utilize the physical capabilities of the original device to create virtual device nodes. For example, using the camera from the original device, a virtual camera node can be created on the Level 1 device, allowing the Level 1 device to receive video data captured by the camera from the original device.

[0069] Secondary devices: These are used to utilize the virtual device capabilities of primary devices to create secondary virtual device nodes. For example, by using the shooting capabilities of the virtual camera on the primary device, a secondary virtual camera node can be created on the secondary device, allowing the secondary device to receive video data captured by the primary device from the camera on the original device.

[0070] In addition to cascading between primary, secondary, and tertiary devices to achieve multi-level virtualization capabilities and sequentially transmit data, at least three devices can also be cascaded for sequential data transmission. This enables the sequential transmission of data collected from the source device across multiple levels of devices.

[0071] The following example, using three cascaded devices with virtual device capabilities as shooting capabilities, illustrates the data transmission scheme provided in this application.

[0072] Figure 1A According to some embodiments of this application, an application scenario diagram is shown where three electronic devices are cascaded to achieve video data sharing.

[0073] like Figure 1A As shown, the application scenario includes electronic device 100, electronic device 200 and electronic device 300. Electronic device 100 is the original device that provides actual shooting capabilities as mentioned above, electronic device 200 is the first-level device that uses the shooting capabilities of the original device, and electronic device 300 is the second-level device that uses the virtual shooting capabilities of the first-level device as mentioned above.

[0074] For ease of explanation, the following description uses a dual-view live streaming scenario where electronic device 100 is drone 100, electronic device 200 is mobile phone 200 as the broadcast assistant, and electronic device 300 is mobile phone 300 as the broadcaster.

[0075] In application scenarios where a drone 100 with superior shooting performance (e.g., a wide field of view) is used, the drone 100 can capture video data with a wide field of view. The drone 100 can provide the mobile phone 200 of the broadcast assistant with the ability to capture video data with a wide field of view, and the mobile phone 200 of the broadcast assistant can provide the mobile phone 300 of the broadcaster with the ability of the drone 100 to capture video data with a wide field of view.

[0076] Specifically, in application scenarios where a drone 100 with superior shooting performance (e.g., a wide field of view) is used to achieve dual-view live streaming (the footage captured by the drone 100 and the footage captured by the streamer's mobile phone 300), for example, such as... Figure 1A As shown, in this application scenario, the drone 100 can capture video data with a wide field of view. The anchor assistant's mobile phone 200 creates a first-level virtual camera from the camera of the drone 100 through a virtualization process. The anchor's mobile phone 300 creates a second-level virtual camera from the virtual camera on the anchor assistant's mobile phone 200 through a second-level virtualization process.

[0077] In this way, the anchor assistant's mobile phone 200 can obtain the video data corresponding to the footage captured by the drone 100. When the anchor assistant is watching the video, the anchor assistant can control the drone 100 by operating the virtual camera of the mobile phone 200, such as adjusting the shooting field of view and shooting angle of the drone 100, so as to select the best footage for the anchor.

[0078] The broadcaster can then use the virtual camera on phone 200 to obtain video data captured by drone 100, while simultaneously using the front-facing camera on phone 300 to film themselves. Phone 300 will then merge the video data obtained from phone 200 via the virtual camera with the image data captured by its own physical camera, enabling dual-view (simultaneous capture of two feeds, such as footage from drone 100 and footage from the broadcaster's phone 300) live streaming.

[0079] As mentioned above, in Figure 1A In the scenario shown, to address the problems in the background art, this embodiment of the application sets up a data transmission path relationship in the drone 100 for sending video data captured by the drone 100 to the mobile phone 200, a data transmission path relationship in the mobile phone 200 for receiving video data sent by the drone 100 providing shooting capabilities, a data transmission path relationship in the mobile phone 200 for sending video data received from the drone 100 to the mobile phone 300, and a data transmission path relationship in the mobile phone 300 for receiving video data sent by the drone 100 from the mobile phone 200. This solves the problem of limitations in the structural performance of the electronic device providing device capabilities (e.g., the drone 100). For example, the electronic device providing device capabilities (e.g., the drone 100) can only connect to one electronic device and cannot simultaneously connect to multiple electronic devices (e.g., the mobile phone 200 and the mobile phone 300) and transmit multiple identical data obtained through device capabilities. For example, when the distance between the electronic device providing device capabilities (e.g., drone 100) and other electronic devices is too far, it is impossible to directly connect the electronic device providing device capabilities (e.g., drone 100) and other electronic devices (mobile phone 300) and provide device capabilities to them through short-range communication technology. This enables multi-level sharing of device capabilities (e.g., mobile phone 200 and mobile phone 300 sharing video data), thereby improving the user experience.

[0080] Figure 1B According to some embodiments of this application, another application scenario diagram is shown where three electronic devices are cascaded to achieve video data sharing.

[0081] like Figure 1BAs shown, the application scenario includes electronic device 100, electronic device 200 and electronic device 300. Electronic device 100 is the original device that provides actual shooting capabilities as mentioned above, electronic device 200 is the first-level device that uses the shooting capabilities of the original device, and electronic device 300 is the second-level device that uses the virtual shooting capabilities of the first-level device as mentioned above.

[0082] For ease of explanation, the following description will use the following example: electronic device 100 is a mining car 100, electronic device 200 is a computer 200 that controls the movement of the mining car 100 to carry out coal mining, and electronic device 300 is a mobile phone 300 that monitors the movement video of the mining car 100 during the coal mining process via the computer 200.

[0083] In this scenario, the mining truck 100, equipped with a camera, is able to capture video data of the path forward for the computer 200, and the computer 200 is able to provide the mobile phone 300 with the ability for the mining truck 100 to capture video data of the path forward for the mobile phone 300.

[0084] Specifically, such as Figure 1B As shown, in this application scenario, on the one hand, if the mining truck 100 simultaneously sends two identical video data streams to the computer 200 and the mobile phone 300, it requires a large amount of bandwidth. Since the mining truck 100 is underground, limited by the poor network conditions, it cannot simultaneously or stably transmit the collected data to both the computer 200 and the mobile phone 300. Therefore, the mining truck 100 is not suitable for simultaneously connecting to the computer 200 and the mobile phone 300 and sending video data to them. On the other hand, due to the limited performance of the mining truck 100, it lacks the capability to send two identical video data streams to both the computer 200 and the mobile phone 300.

[0085] Therefore, computer 200 creates a first-level virtual camera from the camera on mining truck 100 through a virtualization process, and mobile phone 300 creates a second-level virtual camera from the virtual camera on computer 200 through a second-level virtualization process.

[0086] In this way, computer 200 can acquire video data captured by minecart 100 on its forward path, and the user of computer 200 can control minecart 100 by operating the virtual camera of computer 200, such as adjusting the direction and speed of minecart 100.

[0087] Users of mobile phone 300 can obtain video data captured by mining truck 100 through the virtual camera of computer 200 and supervise mining truck 100 during the coal mining process.

[0088] As mentioned above, in Figure 1BIn the scenario shown, to address the problems in the background art, this application embodiment sets up a data transmission path in the mining truck 100 to send video data captured by the mining truck 100 to the computer 200, sets up a data transmission path in the computer 200 to receive video data sent by the mining truck 100 which provides shooting capabilities, sets up a data transmission path in the computer 200 to send video data received from the mining truck 100 to the mobile phone 300, and sets up a data transmission path in the mobile phone 300 to receive video data sent by the mining truck 100 from the computer 200.

[0089] This solves the problem that, due to the structural and performance limitations of the electronic device providing device capabilities (e.g., the mining cart 100), it can only connect to one electronic device at a time, and cannot simultaneously connect to and transmit multiple identical data streams obtained through device capabilities to multiple electronic devices (e.g., the computer 200 and the mobile phone 300). Alternatively, it addresses the issue where the distance between the electronic device providing device capabilities (e.g., the mining cart 100) and other electronic devices (e.g., the mobile phone 300) is too great to directly connect and provide device capabilities via short-range communication technology. This enables multi-level sharing of device capabilities (e.g., the computer 200 and the mobile phone 300 sharing video data), improving the user experience.

[0090] It is understood that the electronic device 100 applicable to the embodiments of this application can be an electronic device that can provide various capabilities (such as shooting capability, sound pickup capability, sound playback capability, etc.) to other devices, such as: camera, mining truck, 5G mining truck, microphone, drone, mobile phone, computer, laptop computer, tablet computer, etc.

[0091] Furthermore, it is understood that the electronic devices 200 and 300 applicable to the embodiments of this application can be electronic devices capable of calling other electronic devices to realize various capabilities, such as mobile phones, computers, laptops, tablets, televisions, vehicle terminals, etc.

[0092] Taking the above three electronic devices as an example of a secondary virtualization scenario, Figure 2According to some embodiments of this application, a system architecture diagram is shown for multi-level wireless transmission of video data between mobile phone 300, mobile phone 200, and drone 100 in the above-described scenario. This system is used to implement the data transmission method of this application. The following mainly describes the topology management module 101 and camera module 103 configured in drone 100, the topology management module 201 and camera virtual module 203 configured in mobile phone 200, and the topology management module 301 and camera virtual module 303 configured in mobile phone 300. Other functional modules in this system will be described when the flowchart is introduced below.

[0093] As mentioned earlier, a camera module 103 can be installed in the drone 100 to provide the function of collecting video data captured by the camera of the drone 100 and sending the video data to the mobile phone 200. A topology management module 101 can be installed in the drone 100 to provide the function of managing data transmission relationships, such as the path relationship for the drone 100 to send video data to the mobile phone 200.

[0094] A virtual camera module 203 can be set up in mobile phone 200 to provide the function of receiving video data captured by the camera of drone 100 and sending the video data to mobile phone 300. A topology management module 201 can be set up in mobile phone 200 to provide the function of managing data transmission relationships, such as the path relationship between mobile phone 200 receiving video data sent by drone 100 and transmitting the received video data to mobile phone 300.

[0095] A virtual camera module 303 can be set up in mobile phone 300 to receive video data captured by the camera of drone 100 sent by mobile phone 200. A topology management module 301 can be set up in mobile phone 300 to provide functions for managing data transmission relationships, such as the path relationship of video data captured by drone 100 sent by mobile phone 200 to mobile phone 300.

[0096] This solves the limitation of existing technologies that only support virtualization between two devices, and enables multiple electronic devices to share video data, thus improving the user experience.

[0097] The following is an example illustrating the software architecture diagram of how a user (e.g., a broadcaster) of mobile phone 300 can obtain video data captured by drone 100 (e.g., a drone) through the virtual camera of mobile phone 200 and use the front camera of mobile phone 300 to take pictures of their own mobile phone 300.

[0098] Figure 3According to some embodiments of this application, a software architecture diagram is shown that enables a user (e.g., a broadcaster) of mobile phone 300 to acquire video data captured by drone 100 (e.g., a drone) through the virtual camera of mobile phone 200, and to capture their own mobile phone 300 using the front-facing camera of mobile phone 300. For example... Figure 3 As shown:

[0099] The software architecture diagram includes the camera driver in the hardware driver layer, the camera module and camera virtualization module in the hardware abstraction layer, the camera framework in the framework layer, and the application (program) in the application layer.

[0100] The hardware driver layer of mobile phone 300 includes a camera driver, which is used to connect to the camera device of mobile phone 300 and drive the camera device of mobile phone 300 to capture images or videos, such as when the user of mobile phone 300 (e.g., a broadcaster) uses the front camera of mobile phone 300 to take pictures of himself.

[0101] The camera driver in the hardware driver layer transmits the images or videos captured by the camera device of the mobile phone 300 sequentially through the camera module in the hardware abstraction layer and the camera frame in the frame layer to the application (program) in the application layer for display.

[0102] The camera virtualization module 303 in the hardware abstraction layer transmits the video received from the mobile phone 200 through the camera frame in the frame layer to the application (program) in the application layer for display.

[0103] The following will be based on Figure 2 The technical solution of this application is described in two stages: establishing a cascading relationship between the drone 100, mobile phone 200 and mobile phone 300, and dismantling the cascading relationship between the drone 100, mobile phone 200 and mobile phone 300.

[0104] The stage of establishing a cascading relationship between drone 100, mobile phone 200, and mobile phone 300:

[0105] Figure 4 According to an embodiment of this application, a schematic diagram of a process for establishing a cascading relationship between a drone 100, a mobile phone 200, and a mobile phone 300 is shown. For example... Figure 4 As shown, it includes:

[0106] First, establish a first-level device virtualization cascading relationship between the drone 100 and the mobile phone 200, as detailed in the following steps.

[0107] 401: After the virtual camera management module 202 of mobile phone 200 and the camera management module 102 of drone 100 discover the device, a connection relationship is established.

[0108] It is understandable that in order for the mobile phone 200 and the drone 100 to transmit commands, camera capability negotiation results, video data, etc., a wired or wireless connection needs to be established between the mobile phone 200 and the drone 100.

[0109] In some embodiments, a wired connection can be established between the mobile phone 200 and the drone 100 via a USB cable or the like.

[0110] In other embodiments, a wireless communication connection can be established between the mobile phone 200 and the drone 100.

[0111] For example, mobile phone 200 and drone 100 log onto the same network. For instance, mobile phone 200 can establish a Wi-Fi connection with drone 100 through a router; or, mobile phone 200 can directly establish a Wi-Fi P2P connection with drone 100. Alternatively, mobile phone 200 and drone 100 can establish a Bluetooth connection.

[0112] Alternatively, mobile phone 200 can directly establish a short-range wireless connection with drone 100. This short-range wireless connection includes, but is not limited to, Near Field Communication (NFC) connection, infrared connection, Ultra Wideband (UWB) connection, and ZigBee connection. Alternatively, mobile phone 200 can directly establish a mobile network connection with drone 100. This mobile network includes, but is not limited to, mobile networks supporting 2G, 3G, 4G, 5G, and subsequent standard protocols. For example, the virtual camera management module 202 of mobile phone 200 sends a wireless connection request to surrounding devices via network broadcast. When drone 100 receives this wireless connection request, its camera management module 102 sends a response to mobile phone 200, thereby establishing a wireless communication relationship between mobile phone 200 and drone 100.

[0113] Alternatively, mobile phone 200 can install an application for managing smart home devices (such as drones, televisions, tablets, air conditioners, speakers, or refrigerators). Taking a smart home application as an example, a user can add one or more smart home devices, establishing a connection between the added devices and mobile phone 200. For instance, a QR code containing device information such as a device identifier can be set on the smart home device. After scanning the QR code with the smart home application on mobile phone 200, the user can add the corresponding smart home device to the application, thereby establishing a connection between the smart home device and mobile phone 200. In this embodiment, when one or more smart home devices added to the smart home application come online, for example, when mobile phone 200 detects a Wi-Fi signal sent by an added smart home device, mobile phone 200 can display the smart home device as a candidate device, prompting the user to select the corresponding smart home device to simultaneously take photos with mobile phone 200.

[0114] Alternatively, if the TV 200 automatically searches for and establishes a wireless communication connection with the drone 100, the mobile phone 300 can directly enter the stage of acquiring video data from the drone 100. In this case, the user does not need to manually select the specific device to establish a wireless communication connection with the mobile phone 200.

[0115] Alternatively, the television 200 may have already established a network connection with one or more electronic devices with camera capabilities. For example, the user may have already established a wireless communication connection with the drone 100 before opening a video call-enabled application on the television 200. Subsequently, the television 200 may no longer need to search for electronic devices with camera capabilities.

[0116] 402: The virtual camera management module 202 of mobile phone 200 sends a configuration command for the virtual camera to the camera virtualization module 203 of mobile phone 200.

[0117] It can be understood that the virtual camera configuration command is used to instruct the mobile phone 200 to call the camera virtualization module 203, that is, to configure the camera virtualization module 203 according to the camera capability negotiation result between the mobile phone 200 and the drone 100. The camera capability negotiation result will be further introduced below.

[0118] 403: The camera virtualization module 203 of mobile phone 200 and the camera proxy module 104 of drone 100 negotiate camera capabilities to obtain the camera capability negotiation result.

[0119] It is understandable that the camera capability negotiation results can include camera configuration parameters. The mobile phone 200 and the drone 100 can negotiate to obtain the best camera configuration parameters.

[0120] Because the mobile phone 200 and the drone 100 process video data in different ways, they need to negotiate their video data processing methods so that the mobile phone 200 can effectively receive the video data transmitted from the drone 100. For example, camera capability negotiation could involve negotiating the video data encoding and decoding capabilities of the drone 100 and the mobile phone 200. For instance, if both the drone 100 and the mobile phone 200 support H.264 and H.265 encoding and decoding capabilities, the negotiation result would be that the drone 200 and the mobile phone 200 would use the better H.265 encoding and decoding capability during video data transmission.

[0121] For example, does the drone 100 perform color space conversion, scaling, rotation, mirroring, filling, image contrast adjustment, and image resolution adjustment on the video data? Does the mobile phone 200 perform color space conversion, scaling, rotation, mirroring, filling, image contrast adjustment, and image resolution adjustment on the video data? The camera capability negotiation result could be that the drone 100 performs color space conversion on the video data, and the mobile phone 200 then performs color space conversion, scaling, rotation, mirroring, and other adjustments on the received video data, but it is not limited to these steps.

[0122] In addition, it is understandable that in order to obtain video data with better playback effect (e.g., image quality), the mobile phone 200 can send user demand information or parameter information stored in itself for configuring the camera of the drone 100 to the drone 100.

[0123] For example, camera capability negotiation can also negotiate camera parameter settings such as camera resolution, frame rate, sensitivity (ISO value), continuous shooting speed, focus speed, exposure time, image bit depth, pixel size, exposure mode, and focus mode in the drone 100.

[0124] 404: The camera virtualization module 203 of mobile phone 200 is configured according to the camera capability negotiation result.

[0125] The camera virtualization module 203 of the mobile phone 200 can be configured according to the camera capability negotiation results of the video data processing method. For example, the mobile phone 200 can configure the camera virtualization module 203 to use the better H.265 codec capability to decode the video data received from the drone 100 during the video data transmission process.

[0126] 405: The camera virtualization module 203 of mobile phone 200 sends the configuration information of the virtual camera to the virtual camera node management module 205 of mobile phone 200.

[0127] It is understandable that the configuration information of the virtual camera can be the configuration time of the camera virtualization module 203 and the camera capability negotiation results used to configure the camera virtualization module 203. This allows the mobile phone 300 to directly obtain the camera capability negotiation results between the mobile phone 200 and the drone 100 from the virtual camera node management module 205 to configure its own camera virtualization module 303 when establishing a data transmission relationship between the mobile phone 300 and the drone 200. This saves time in establishing a data transmission relationship between the mobile phone 300 and the drone 200 and improves the efficiency of establishing such a relationship.

[0128] 406: The virtual camera node management module 205 of mobile phone 200 sends a notification that the virtual camera has been successfully configured to the virtual camera management module 202 of mobile phone 200.

[0129] 407: After receiving the notification from the virtual camera node management module 205 of mobile phone 200 that the virtual camera configuration was successful, the virtual camera management module 202 of mobile phone 200 sends an update data transmission relationship notification to the topology management module 201 of mobile phone 200.

[0130] The topology management module 101 is used to manage the data transmission relationships between multiple levels of devices. Specifically, managing the data transmission relationships between multiple levels of devices can include updating the data transmission relationships between devices: adding device nodes in the data transmission relationship, removing device nodes from the data transmission relationship, etc.

[0131] In this embodiment of the application, before the virtual camera management module 202 of the mobile phone 200 receives the notification from the virtual camera node management module 205 of the mobile phone 200 that the configuration of the virtual camera is successful, the data transmission relationship is that the drone 100 has no video data transmission object.

[0132] After the virtual camera management module 202 of mobile phone 200 receives the notification from the virtual camera node management module 205 of mobile phone 200 that the virtual camera configuration was successful, the data transmission relationship is that the drone 100 transmits the video data captured by the camera to mobile phone 200.

[0133] Therefore, it can be understood that the update data transmission relationship notification includes a new data transmission relationship for the drone 100 to transmit video data captured by the camera to the mobile phone 200.

[0134] 408: After receiving the update data transmission relationship notification sent by the virtual camera management module 202 of the mobile phone 200, the topology management module 201 of the mobile phone 200 sends the update data transmission relationship notification to the topology management module 101 of the drone 100.

[0135] The update data transmission relationship notification includes a new data transmission relationship for drone 100 to transmit video data captured by its camera to mobile phone 200.

[0136] Data transmission relationships can be recorded in the following format:

[0137] {

[0138] SourceDevice: Drones 100

[0139] Capability: camera

[0140] Virtualization Path: Drones 100, Mobile Phones 200

[0141] }

[0142] Here, SourceDevice represents the source device providing the device capabilities; the content following "SourceDevice:" indicates that the source device providing the device capabilities is the drone 100. Capability represents the device capabilities provided by the source device; the content following "Capability:" indicates that the device capability provided by the source device is a shooting capability. Virtualization Path represents the data transmission relationship; the content following "Virtualization Path:" indicates that the drone 100 transmits video data captured by its camera to the mobile phone 200.

[0143] After establishing a primary device virtualization cascading relationship between drone 100 and mobile phone 200, a secondary device virtualization cascading relationship is established between mobile phone 200 and mobile phone 300. See the following steps for details.

[0144] 409: After device discovery, the virtual camera management module 302 of mobile phone 300 and the virtual camera management module 202 of mobile phone 200 establish a connection relationship.

[0145] Step 409 and step 401 above are based on the same inventive concept, and will not be repeated here.

[0146] 410: The virtual camera management module 202 of mobile phone 200 sends a connection success message to the camera virtualization module 203 of mobile phone 200.

[0147] 411: After receiving the connection success message sent by the virtual camera management module 202 of the mobile phone 200, the camera virtualization module 203 of the mobile phone 200 sends a query virtual capability request to the virtual camera management module 202 of the mobile phone 200.

[0148] Mobile phone 200 may simultaneously use the shooting, sound pickup, and sound playback capabilities of an external device. In order to determine whether mobile phone 200 is using the shooting capability of an external device, after receiving the connection success information sent by the virtual camera management module 202 of mobile phone 200, the camera virtualization module 203 of mobile phone 200 sends a query virtual capability request to the virtual camera management module 202 of mobile phone 200. The query virtual capability request is used to query the capabilities of the external device used by mobile phone 200.

[0149] 412: The virtual camera management module 202 of mobile phone 200 sends the virtual capability as virtual shooting capability to the virtual camera management module 302 of mobile phone 300.

[0150] 413: When the virtual camera management module 302 of mobile phone 300 receives that the virtual capability of mobile phone 200 is virtual shooting capability, the virtual camera management module 302 of mobile phone 200 sends a configuration virtual camera instruction to the camera virtualization module 303 of mobile phone 300.

[0151] 414: The camera virtualization module 303 of mobile phone 300 and the virtual camera proxy module 204 of mobile phone 200 negotiate camera capabilities to obtain the camera capability negotiation result.

[0152] The specific negotiation process is as follows:

[0153] 414a: The virtual camera proxy module 204 of mobile phone 200 sends a request to the virtual camera node management module 205 of mobile phone 200 to query the virtual camera capability.

[0154] Since the mobile phone 200 and the drone 100 have already negotiated camera capabilities once in step 403, the virtual camera capability request is queried in order to obtain the camera capability negotiation result of step 403.

[0155] 414b: The virtual camera proxy module 204 of mobile phone 200 receives the queried virtual camera capability information sent by the virtual camera node management module 205 of mobile phone 200.

[0156] The virtual camera capability information can be the result of camera capability negotiation. See step 403 for details on the camera capability negotiation result, which will not be repeated here.

[0157] 414c: The virtual camera proxy module 204 of the mobile phone 200 sends a camera authorization request of the drone 100 to the camera management module 102 of the drone 100.

[0158] To ensure the security of video data transmission captured by the drone 100, the mobile phone 200 provides shooting functionality to the mobile phone 300 only after the drone 100 grants control to the mobile phone 300. That is, the mobile phone 200 can only transmit data received from the drone 100 to the mobile phone 300 after authorization from the drone 100.

[0159] For example, Figure 5 According to some embodiments of this application, a schematic diagram of the control and data transmission authorization process of a camera in a drone 100 is shown, such as... Figure 5 As shown, the display interface of mobile phone 200 displays a prompt box 51, which includes the prompt message: "Do you want to send a request to the drone to grant mobile phone 300 control of the drone's camera and the right to acquire video data?", as well as "Yes" control 511 and "No" control 512 for the user to select.

[0160] After the user selects (e.g., clicks) the "Yes" control 511, the mobile phone 200 detects this operation and sends an authorization request to the drone 100.

[0161] 414c: The camera management module 102 of the drone 100 determines whether authorization is required. If yes, it proceeds to 414e. If no, the camera management module 102 of the drone 100 sends an unauthorized camera instruction to the virtual camera proxy module 204 of the mobile phone 200.

[0162] Please continue reading. Figure 5 After receiving the authorization request from the drone 100, the mobile phone 400 controlling the drone 100 displays a prompt box 52 on its screen. The prompt box 52 includes the message: "Do you grant control of the drone's camera and the right to acquire video data to the mobile phone 300?", and "Yes" and "No" controls 521 and 522 for the user to choose from. If the user selects "Yes" control 521, the camera management module 102 of the drone 100 determines that it has received authorization feedback from the mobile phone 400 and proceeds to 414e. If the user selects "Yes" control 522, the camera management module 102 of the drone 100 sends an unauthorized camera indication to the virtual camera proxy module 204 of the mobile phone 200.

[0163] It is understandable that, in addition to mobile phones 400, other control terminals such as computers and remote controls can be used to control the drone 100, and no restrictions are imposed here.

[0164] Furthermore, it is understood that the nature of the authorization can be temporary (e.g., three days) or permanent.

[0165] 414e: The camera management module 102 of the drone 100 sends a camera authorization instruction of the drone 100 to the virtual camera proxy module 204 of the mobile phone 200.

[0166] 415: The camera virtualization module 303 of mobile phone 300 is configured according to the camera capability negotiation result.

[0167] Steps 415 and 404 are based on the same inventive concept, and will not be described again here.

[0168] 416: The camera virtualization module 303 of mobile phone 300 sends the configuration information of the virtual camera to the device node management module 305 of mobile phone 300.

[0169] The configuration information of the virtual camera can be the configuration time of the camera virtualization module 303 and the negotiation result of the camera capabilities used to configure the camera virtualization module 303.

[0170] 417: The device node management module 305 of mobile phone 300 sends a notification that the virtual camera has been successfully configured to the virtual camera management module 302 of mobile phone 300.

[0171] Before the device node management module 305 of mobile phone 300 sends a notification that the virtual camera has been successfully configured to mobile phone 300 to the virtual camera management module 302 of mobile phone 300, the data transmission relationship is that the drone 100 transmits video data captured by the camera to mobile phone 200.

[0172] After the device node management module 305 of mobile phone 300 sends a notification that the virtual camera has been successfully configured to the virtual camera management module 302 of mobile phone 300, the data transmission relationship is that the drone 100 transmits video data captured by the camera to mobile phone 200, and mobile phone 200 transmits the video data received from drone 100 back to mobile phone 300.

[0173] Therefore, it can be understood that the update data transmission relationship notification includes a new data transmission relationship in which the drone 100 transmits video data captured by its camera to the mobile phone 200, and the mobile phone 200 then transmits the video data received from the drone 100 to the mobile phone 300.

[0174] Data transmission relationships can be recorded in the following format:

[0175] {

[0176] SourceDevice: Drones 100

[0177] Capability: camera

[0178] Virtualization Path: Drones 100, Phones 200, Phones 300

[0179] }

[0180] 418: After receiving the notification from the device node management module 305 of the mobile phone 300 that the virtual camera configuration was successful, the virtual camera management module 302 of the mobile phone 300 sends an update data transmission relationship notification to the topology management module 301 of the mobile phone 300.

[0181] It is understandable that the update data transmission relationship notification includes a new data transmission relationship whereby the drone 100 transmits video data captured by its camera to the mobile phone 200, and the mobile phone 200 then transmits the video data received from the drone 100 to the mobile phone 300.

[0182] 419: After receiving the update data transmission relationship notification sent by the virtual camera management module 302 of the mobile phone 300, the topology management module 301 of the mobile phone 300 sends the update data transmission relationship notification to the topology management module 201 of the mobile phone 200.

[0183] It is understandable that the update data transmission relationship notification includes a new data transmission relationship whereby the drone 100 transmits video data captured by its camera to the mobile phone 200, and the mobile phone 200 then transmits the video data received from the drone 100 to the mobile phone 300.

[0184] 420: After receiving the update data transmission relationship notification sent by the virtual camera management module 302 of the mobile phone 300, the topology management module 201 of the mobile phone 200 sends the update data transmission relationship notification to the topology management module 101 of the drone 100.

[0185] It is understandable that the update data transmission relationship notification includes a new data transmission relationship whereby the drone 100 transmits video data captured by its camera to the mobile phone 200, and the mobile phone 200 then transmits the video data received from the drone 100 to the mobile phone 300.

[0186] Existing technologies do not support multi-level device virtualization, thus eliminating the need for multi-level device data transmission relationship management. However, in this embodiment, based on support for cascading virtualization, users can view the current data transmission relationship and manage devices within the relationship according to actual needs (cascading continuation and de-cascading). De-cascading allows devices participating in the data transmission relationship to exit; otherwise, as the number of devices increases, the difficulty of business implementation and the end-user experience will continuously deteriorate. Cascading continuation allows devices not currently participating in the data transmission relationship to join it.

[0187] Based on the implementation of the above-mentioned distributed capability virtualization, in the application phase:

[0188] When mobile phone 300 detects a user operation invoking distributed capability virtualization, it sends a call command to mobile phone 200. Mobile phone 200 then sends a call command to the drone based on this command. The drone 100 can then provide shooting functionality to mobile phone 200 according to the call command. In other words, mobile phone 200 can control the drone 100's operating status and / or data transmission relationships. For example, the drone 100 can send video data collected through shooting to mobile phone 200. Alternatively, mobile phone 200 can send control commands to the drone 100 to adjust its shooting field of view and shooting angle, etc.

[0189] Mobile phone 200 provides the drone's shooting function to mobile phone 300 based on the calling command sent by mobile phone 300. That is, mobile phone 300 can control the working status and / or data transmission status of drone 100 through mobile phone 200. For example, mobile phone 200 sends video data received from drone 100 to mobile phone 300. Alternatively, mobile phone 300 can send control commands to drone 100 through mobile phone 200 to adjust the drone 100's shooting field of view and shooting angle, etc. Furthermore, mobile phone 300 can send control commands to mobile phone 200 to control the video data transmission status of mobile phone 200, such as pausing video data transmission or disabling the video data transmission function.

[0190] Based on the above-mentioned distributed capability virtualization, this application provides two ways to de-cascade. One is that the source device initiates the de-cascade process: that is, the original device that provides the original physical device capability initiates the de-virtualization process, and then the devices in the entire virtualization chain complete the de-virtualization capability one by one, and finally all exit virtualization.

[0191] Figure 6 According to some embodiments of this application, a schematic diagram of a cascading release process initiated by a drone 100 is shown. Figure 6 As shown, it includes:

[0192] 601: The topology management module 101 of the drone 100 sends a command to the camera management module 102 of the drone 100 to end the virtual camera.

[0193] It is understandable that after the above-mentioned distributed capability virtualization is implemented, users can query the data transmission relationship on drone 100, mobile phone 200 and mobile phone 300, and dynamically remove devices participating in the data transmission relationship according to user needs. In the event that an intermediate device withdraws from the data transmission relationship, a data transmission relationship will be established between the two devices connected to the intermediate device, thereby ensuring the continuity of data transmission services.

[0194] 602: The camera management module 102 of the drone 100 sends a command to the virtual camera management module 202 of the mobile phone 200 to end the virtual camera.

[0195] 603: The virtual camera management module 202 of mobile phone 200 sends a command to the virtual camera management module 302 of mobile phone 300 to end the virtual camera, with the reason being that it was initiated by the source device drone 100.

[0196] 604: The virtual camera management module 202 of mobile phone 200 sends an update data transmission relationship notification to the topology management module 201 of mobile phone 200.

[0197] It is understandable that the updated data transmission relationship notification indicates that the data relationship is empty, meaning that the video data transmission relationship between drone 100, mobile phone 200, and mobile phone 300 has ended.

[0198] 605: The virtual camera management module 302 of mobile phone 300 sends an update data transmission relationship notification to the topology management module 301 of mobile phone 300.

[0199] It is understandable that the updated data transmission relationship notification indicates that the data relationship is empty, meaning that the video data transmission relationship between drone 100, mobile phone 200, and mobile phone 300 has ended.

[0200] 606: The topology management module 301 of mobile phone 300 sends an update data transmission relationship notification to the topology management module 201 of mobile phone 200.

[0201] It is understandable that the updated data transmission relationship notification indicates that the data relationship is empty, meaning that the video data transmission relationship between drone 100, mobile phone 200, and mobile phone 300 has ended.

[0202] 607: The topology management module 201 of mobile phone 200 sends an update data transmission relationship notification to the topology management module 101 of drone 100.

[0203] It is understandable that the updated data transmission relationship notification indicates that the data relationship is empty, meaning that the video data transmission relationship between drone 100, mobile phone 200, and mobile phone 300 has ended.

[0204] Another approach is to initiate termination and then reconnection via intermediate device: this involves only terminating the virtualization capabilities of the intermediate device while simultaneously connecting the upstream and downstream devices to continue virtualizing device capabilities. For example, a secondary device (phone 300) initiates termination of virtualization for a primary device (phone 200) and simultaneously connects the source device (drone 100) to the secondary device (phone 300). While compressing the primary device, this ensures that the virtualization services between the original secondary device (phone 300) and the new primary device (phone 200) are not interrupted. Compared to the scenario where only the virtualization capabilities of the intermediate device are terminated, which requires a series of operations such as re-authentication to establish a communication connection between the secondary device (phone 300) and the source device (drone 100), this simplifies the process of establishing a virtualization capability communication relationship between the secondary device (phone 300) and the source device (drone 100).

[0205] Figure 7 According to some embodiments of this application, a schematic diagram of a cascading termination process initiated by an intermediate device, mobile phone 200, is shown. For example... Figure 7 As shown, it includes:

[0206] 701: The topology management module 301 of mobile phone 300 sends a command to the virtual camera management module 302 of mobile phone 300 to terminate the data transmission relationship between mobile phone 300 and mobile phone 200.

[0207] 702: The virtual camera management module 302 of mobile phone 300 sends a command to the virtual camera management module 202 of mobile phone 200 to terminate the data transmission relationship between mobile phone 300 and mobile phone 200.

[0208] 703: The virtual camera management module 202 of mobile phone 200 sends the device information of drone 100 to the virtual camera management module 302 of mobile phone 300.

[0209] It is understandable that after the data transmission relationship between mobile phone 200 and mobile phone 300 for sending video data captured by the drone is terminated, mobile phone 300 can establish a data transmission relationship with the drone that provides shooting capability to mobile phone 200. Therefore, mobile phone 200 sends the device information of drone 100 to mobile phone 300 to facilitate the re-establishment of the data transmission relationship between mobile phone 300 and drone 100.

[0210] The device information of the drone 100 can include the IP address, MAC address, device ID, and the device connection method between the drone 100 and the mobile phone 200 (e.g., Wi-Fi connection, Wi-Fi P2P connection, Bluetooth connection, or short-range wireless connection) that enable the establishment of a wireless communication connection between the drone 100 and the mobile phone 200.

[0211] 704: The virtual camera management module 302 of mobile phone 300 and the camera management module 102 of drone establish a connection relationship after device authentication.

[0212] It is understandable that since the mobile phone 300 has obtained the device information of the drone 100 that provides shooting capabilities, the mobile phone 300 and the drone 100 can establish a wireless connection relationship through short-range communication technology (such as WIFI, Bluetooth, etc.). For example, the virtual camera management module 302 of the mobile phone 300 directly sends a wireless connection request to the drone 100, and the camera management module 102 of the drone 100 sends a response feedback to the mobile phone 200, thereby enabling the mobile phone 200 and the drone 100 to establish a wireless communication relationship.

[0213] 705: The virtual camera management module 302 of the mobile phone 300 sends an update command to the camera virtualization module 303 of the mobile phone 300.

[0214] It can be understood that the virtual camera configuration command is used to instruct the mobile phone 300 to update the camera virtualization module 303, that is, to configure the camera virtualization module 303 according to the camera capability negotiation result between the mobile phone 300 and the drone 100. The camera capability negotiation result will be further introduced below.

[0215] 706: The camera virtualization module 303 of the mobile phone 300 and the camera proxy module 104 of the drone negotiate device capabilities and obtain the device capability negotiation result.

[0216] The technical means of step 706 are the same as those of step 403, and will not be described again here.

[0217] 707: The camera virtualization module 303 of the mobile phone 300 updates the virtual camera according to the negotiation results of the device capabilities.

[0218] The technical means of step 707 are the same as those of step 404, and will not be described again here.

[0219] 708: The camera virtualization module 303 of mobile phone 300 sends the update information of the virtual camera to the device node management module 305 of mobile phone 300.

[0220] It is understandable that the configuration information of the virtual camera can include the configuration time of the camera virtualization module 303 and the camera capability negotiation results used to configure the camera virtualization module 303. This allows other devices to directly obtain the camera capability negotiation results between the mobile phone 300 and the drone 100 from the virtual camera node management module 305 to configure their camera virtualization modules, saving time and improving the efficiency of establishing data transmission relationships between other devices and the mobile phone 300.

[0221] 709: The camera virtualization module 303 of mobile phone 300 sends a notification that the virtual camera has been successfully updated to the virtual camera management module 302 of mobile phone 300.

[0222] It is understandable that the update data transmission relationship notification includes a new data transmission relationship for the drone 100 to transmit video data captured by its camera to the mobile phone 300.

[0223] 710: The virtual camera management module 302 of mobile phone 300 sends an update data transmission relationship notification to the topology management module 301 of mobile phone 300.

[0224] It is understandable that the update data transmission relationship notification includes a new data transmission relationship for the drone 100 to transmit video data captured by its camera to the mobile phone 300.

[0225] After the cascading undoing and reconnection process is completed, the updated data transmission relationship can be recorded in the following format:

[0226] {

[0227] SourceDevice: Drones 100

[0228] Capability: camera

[0229] Virtualization Path: Drones 100, Mobile Phones 300

[0230] }

[0231] 711: The topology management module 301 of mobile phone 300 sends an update data transmission relationship notification to the topology management module 101 of drone 100.

[0232] It is understandable that the update data transmission relationship notification includes a new data transmission relationship for the drone 100 to transmit video data captured by its camera to the mobile phone 300.

[0233] Figure 8According to some embodiments of this application, a structural schematic diagram of a drone 100 that implements the technical solution of this application is shown.

[0234] like Figure 8 The diagram shown is a structural schematic of a drone 100 disclosed according to an embodiment of this application. The drone 100 may include a communication module 1001, a camera 1002, a processor 1003, and a power module 1004, etc.

[0235] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the drone 100. In other embodiments of this application, the drone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0236] The communication module 1001 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 1001 can provide solutions for wireless communication applied to the drone 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The drone 100 can communicate with networks and other devices via wireless communication technologies. In this embodiment, the drone 100 can communicate with a mobile phone 200 through the communication module 1001, sending captured video data to the mobile phone 200.

[0237] Processor 1003 may include one or more processing units, such as processing modules or circuits of CPU, ISP, graphics processing unit (GPU), DSP, micro-programmed control unit (MCU), artificial intelligence (AI) processor, or field-programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 1002 may include memory units for storing instructions and data.

[0238] Camera 1002 is used to capture still images or videos. In the embodiment of the application, camera 1002 is used to collect video data.

[0239] The power module 1004 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 1002. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 1003, the camera 1002, and the communication module 1001, etc.

[0240] Figure 9 According to some embodiments of this application, a structural schematic diagram of a mobile phone 200 implementing the technical solution of this application is shown. For example... Figure 9 As shown:

[0241] Mobile phone 200 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an eSIM card 196, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0242] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 200. In other embodiments of this application, the mobile phone 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0243] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0244] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0245] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0246] The wireless communication function of mobile phone 200 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. In this embodiment, mobile phone 200 can interact with drone 100 through these modules to obtain video data from drone 100. Mobile phone 200 can also interact with mobile phone 300 to send video data to mobile phone 300.

[0247] The mobile phone 200 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0248] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a Micro LED, a Micro OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, mobile phone 200 may include one or N displays 194, where N is a positive integer greater than 1. In this embodiment, video can be displayed based on video data acquired from drone 100.

[0249] The mobile phone 200 can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0250] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile phone 200. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0251] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of mobile phone 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of mobile phone 200 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0252] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with or separate from the mobile phone 200. The mobile phone 200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards.

[0253] In some embodiments, the mobile phone 200 further includes an eSIM card 196, i.e., an embedded SIM card. The eSIM card 196 can be embedded in the mobile phone 200 and cannot be separated from the mobile phone 200. The eSIM card 196 can be integrated into the mobile phone's system-on-chip (SOC) chip, modem chip, or near field communication (NFC) chip, and this application embodiment does not impose any limitations on this.

[0254] The software system of mobile phone 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of mobile phone 200.

[0255] Figure 10 According to some embodiments of this application, a software structure block diagram of a mobile phone 200 is shown.

[0256] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries (which can also be the hardware abstraction layer), and the kernel layer.

[0257] The application layer can include a series of application packages. For example... Figure 10 As shown, the application package can include applications such as shopping, news, camera, gallery, calendar, calling, maps, navigation, WLAN, Bluetooth, music, video, and SMS. The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0258] like Figure 10 As shown, the application framework layer may include a window manager, page provider, view system, phone manager, resource manager, notification manager, etc.

[0259] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0260] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0261] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0262] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0263] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0264] The system library can include multiple functional modules. For example: virtual camera module, surface manager, media libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0265] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0266] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0267] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for distributed device capability virtualization, the method comprising: include: A first-level communication relationship is established between a first electronic device and a second electronic device, wherein the first-level communication relationship is used to instruct the first electronic device to provide a preset capability to the second electronic device, the preset capability including a shooting capability; wherein the preset capability of the first electronic device is controlled based on the operation of the second device. The third electronic device sends a request to the second electronic device to establish a second-level communication relationship; wherein, the second-level communication relationship request is used by the third electronic device to obtain authorization from the second electronic device to utilize the preset capability through the second electronic device; When the second electronic device requests permission from the third electronic device to establish a second-level communication relationship with the second electronic device based on the second-level communication relationship request, a second-level communication relationship is established between the third electronic device and the second electronic device, wherein the second-level communication relationship is used to instruct the third electronic device to utilize the preset capability through the second electronic device; The third electronic device merges the image data acquired by the third electronic device with the image data acquired using the preset capability.

2. The method of claim 1, wherein, The method further includes: when the third electronic device sends a first calling instruction to the second electronic device, and the second electronic device sends a second calling instruction to the first electronic device according to the first calling instruction; The first electronic device provides the preset capability to the second electronic device according to the second invocation instruction; The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction.

3. The method according to claim 2, characterized in that, The first electronic device provides the preset capability to the second electronic device according to the second invocation instruction; The second electronic device provides a preset capability to the third electronic device according to the first invocation instruction, including: the first electronic device sending first data generated to the second electronic device to realize the preset capability; The second electronic device sends the first data received from the first electronic device to the third electronic device.

4. The method according to claim 2, characterized in that, The first electronic device provides the preset capability to the second electronic device according to the second invocation instruction; The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction, including: The third electronic device sends second data generated to the second electronic device to realize the preset capability; The second electronic device sends the second data received from the third electronic device to the first electronic device.

5. The method according to claim 1, characterized in that, The preset capabilities include one or more of the following: audio playback capability and audio acquisition capability.

6. The method of claim 1, wherein, The method further includes: when the third electronic device obtains an instruction to establish a third-level communication relationship with the first electronic device, and the third electronic device sends an instruction to terminate the second-level communication relationship to the second electronic device according to the instruction to establish a third-level communication relationship, the second electronic device sends the device information of the first electronic device to the third electronic device according to the instruction to terminate the second-level communication relationship and closes the second-level communication relationship; The third electronic device establishes a third-level communication relationship with the first electronic device based on the device information of the first electronic device, wherein the third-level communication relationship is used to instruct the first electronic device to provide a preset capability to the third electronic device.

7. A method for distributed device capability virtualization, the method comprising: include: The second electronic device receives a request from the third electronic device to establish a second-level communication relationship; When the second electronic device requests permission from the third electronic device to establish a second-level communication relationship with the second electronic device based on the second-level communication relationship request, a second-level communication relationship is established between the third electronic device and the second electronic device. This second-level communication relationship instructs the third electronic device to utilize preset capabilities of the first electronic device through the second electronic device, including imaging capabilities. When the second electronic device terminates the first-level communication relationship with the first electronic device, the third electronic device establishes a third-level communication relationship with the first electronic device. The third-level communication relationship is used to instruct the first electronic device to provide the preset capability to the third electronic device.

8. The method of claim 7, wherein, The method further includes: the second electronic device receiving a first invocation instruction sent by the third electronic device, and the second electronic device sending a second invocation instruction to the first electronic device according to the first invocation instruction; The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction.

9. The method of claim 8, wherein, The second electronic device provides a preset capability to the third electronic device according to the first invocation instruction, including: the second electronic device sending the first data received from the first electronic device to the third electronic device.

10. The method of claim 8, wherein, The second electronic device provides preset capabilities to the third electronic device according to the first invocation instruction, including: The second electronic device sends the second data received from the third electronic device to the first electronic device.

11. The method according to claim 7, characterized in that, The preset capabilities include one or more of the following: audio playback capability and audio acquisition capability.

12. The method according to claim 7, characterized in that, The method further includes: when the third electronic device obtains an instruction to establish a third-level communication relationship with the first electronic device, and the third electronic device sends an instruction to terminate the second-level communication relationship to the second electronic device according to the instruction to establish a third-level communication relationship, the second electronic device sends the device information of the first electronic device to the third electronic device according to the instruction to terminate the second-level communication relationship and closes the second-level communication relationship.

13. A readable medium characterized by The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to implement the distributed device capability virtualization method according to any one of claims 1 to 12.

14. An electronic device, characterized in that, include: Memory, used to store instructions; as well as A processor for executing the instructions to implement the distributed device capability virtualization method according to any one of claims 1 to 12.

Images