WebRTC-based video assistance and device remote calling method and device

Abstract

The invention relates to the technical field of remote control, in particular to a WebRTC-based video assistance and equipment remote calling method and device. Comprising the following steps that: a WebRTC connection is established, and a local terminal initiates an equipment information acquisition request to a remote terminal; after the remote terminal receives the equipment information acquisition request, the equipment information is collected through the hardware peripheral manager and is returned to the remote terminal; after receiving the equipment information, the local terminal performs interface compatibility check on hardware peripheral calling interface information contained in the equipment information, if an interface is not compatible, a user is prompted to perform version matching upgrading, and otherwise, the next step is performed; after the local terminal completes equipment information checking, checking whether a remote hardware peripheral calling plug-in of a corresponding version is installed locally; according to the device, the data types output by the block device and the stream device can be well supported at the same time, and better performance and interaction experience are provided.

Description

Technical Field

[0001] The present invention relates to the technical field of remote control, and in particular to a method and device for video assistance and remote device calling based on WebRTC. Background Art

[0002] In fields such as education and government affairs, there are often scenarios of remote assistance and remote operation, which include scenarios where it is necessary to call the hardware peripherals of a remote terminal, collect relevant data, and then transmit and enter it into the local terminal business system. The current technical solutions for realizing such remote call scenarios mainly include remote desktop and proxy programs based on the B / S architecture. The implementation method of remote desktop is to open and run the remote desktop client program in the remote terminal system, and locally, through the remote desktop control program installed in the local terminal system, establish a connection with the remote desktop client program on the other end through the server and realize remote control. Through the remote control method, run the business system on the remote terminal, call the hardware peripherals of the remote terminal, collect relevant data and enter it into the business system.

[0003] In the above traditional remote scenario call technical solutions, it is necessary to run the local business system on the remote terminal. If, due to various conditions, it is impossible to run the local business system on the remote terminal, it is necessary to develop an additional proxy system to be responsible for calling the hardware peripherals and collecting the corresponding data, and sending it to the local business system through a transfer method.

[0004] Based on the above problems, there is an urgent need for a WebRTC-based video assistance and remote device calling system, which does not require additional development of a proxy system even if it is impossible to run the local business system on the remote terminal. Summary of the Invention

[0005] In view of the current remote scenario call technical solutions, it is impossible to run the local business system on the remote terminal and it is necessary to develop an additional proxy system. The present application provides a method and device for video assistance and remote device calling based on WebRTC to solve the above problems.

[0006] To achieve the above object, the present invention is realized through the following technical solutions: An embodiment of the present application discloses a method for video assistance and remote device calling based on WebRTC, including: After establishing that the WebRTC connection is ready, the local terminal sends a request to the remote terminal to obtain device information; after receiving the request to obtain device information, the remote terminal collects the device information through the hardware peripheral manager and returns it to the remote terminal; After the local terminal receives the device information, it checks the interface compatibility of the hardware peripheral call interface information contained therein. If the interface is not compatible, it prompts the user to perform version matching and upgrade, otherwise it proceeds to the next step; After the local terminal completes the device information check and verification, it checks whether the corresponding version of the remote hardware peripheral call plug-in has been installed locally; if the corresponding version of the plug-in has been installed, it proceeds to the step of creating a remote hardware peripheral object, otherwise it proceeds to the next step; The local terminal sends a request to obtain a plug-in for a remote hardware peripheral to the remote terminal; after receiving the request, the remote terminal packages the plug-in based on HTML+CSS+JavaScript and sends it to the local terminal through the WebRTC data channel; After receiving the plug-in, the local terminal verifies, unpacks, installs, loads and registers the plug-in, then creates the corresponding remote hardware peripheral object and hands it over to the abstract hardware peripheral manager for management and scheduling; When the business system initiates a device call request, and the user directs the request to a remote hardware peripheral object through selection or configuration, the remote hardware peripheral object is responsible for encapsulating the call request and the corresponding related parameters and sending them to the remote terminal; After receiving the call request, the remote terminal transfers the request flow to the remote call processor for analysis, and then assigns it to the hardware peripheral manager. The hardware peripheral manager schedules and drives the corresponding hardware peripherals to collect data, and finally the remote call processor transmits the collected data to the local terminal; After the local terminal receives the collected data, it processes the collected data and finally submits it to the business system.

[0007] A device, applied to a video assistance and device remote calling method based on WebRTC as described above, characterized in that it comprises: a browser container device end and a WebRTC technology end, wherein the browser container device end comprises: Hardware peripheral calling module: The hardware peripheral calling module can drive the hardware peripherals on the terminal, initiate calls and collect corresponding data; Hardware peripheral manager module: The hardware peripheral manager module can manage various hardware peripherals integrated in the terminal, and uniformly allocate and schedule them; Abstract hardware peripheral manager module: The browser container device has the ability to abstractly manage hardware peripherals. In addition to managing hardware peripherals on the local terminal, it can also abstract and manage hardware peripherals on remote terminals, and uniformly allocate and schedule them. Hardware peripheral calling interface object module: The browser container device provides the business system with a corresponding unified hardware peripheral interface object according to the type of hardware peripherals. The business system can drive the hardware peripherals and collect corresponding data by calling the interface object; Remote Call Interface Module: The remote call interface abstracts the remote call method and parameters to form a unified call interface, enabling different network communication methods to be called in a unified manner; Remote Call Processor Module: The remote call manager on the browser container device side is mainly responsible for parsing, allocating, and responding to remote call requests; Plugin Extension Module: The browser container provides a plugin extension mechanism based on HTML+CSS+JavaScript technology, supporting functions such as plugin verification, installation, loading, registration, management, and uninstallation.

[0008] Preferably, the multimedia channel of the WebRTC technology side can be used to transmit the output data of the streaming device, while the data channel can be used to transmit the output data of the block device.

[0009] More preferably, after receiving a call request, the Hardware Peripheral Call Interface Object Module encapsulates the call request and corresponding relevant parameters, and then sends them to the Abstract Hardware Peripheral Manager, which schedules the hardware peripheral object according to user selection or configuration.

[0010] More preferably, the Remote Call Processor Module receives and parses the hardware peripheral remote call request from the peer end, then allocates the request to the Abstract Hardware Peripheral Manager, which is responsible for driving the call to the local hardware peripheral and collecting the corresponding data. Finally, the Remote Call Processor encapsulates the result data and responds back to the peer end.

[0011] More preferably, the Plugin Extension Module depends on this plugin extension mechanism, dynamically installs and registers remote hardware peripheral plugins, creates remote hardware peripheral objects, and hands them over to the Abstract Hardware Peripheral Manager for management.

[0012] Technical Effects: 1. The technical solution of the present invention is based on the WebRTC technology. Through the data channel and multimedia stream channel of WebRTC, it can well support the data types output by both block devices and streaming devices, and provide better performance and interaction experience; 2. Provide more flexible call methods and interaction methods. The technical solution of the present invention is based on the plugin extension mechanism of the browser container and the implementation method of its plugins through the HTML+CSS+JavaScript technology stack. It can not only provide the background interface call method but also provide the interaction method of the user interface, and can flexibly adapt to the characteristics of hardware peripherals and meet the needs of users and business systems.

[0013] 3. In the technical solution of the present invention, the remote hardware peripheral object realizes the decoupling of the remote call of the hardware peripheral and the network communication by inheriting and implementing the hardware peripheral interface and abstracting and encapsulating the network communication, so that the technical solution of the present invention can not only support WebRTC, but also support network communication methods such as WebSocket and HTTP. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a flowchart of the video assistance and device remote call method based on WebRTC of the present application; Figure 2 It is a connection block diagram of the video assistance and device remote call method based on WebRTC of the present application.

[0016] In the figure: 1. Hardware peripheral call module; 2. Hardware peripheral manager module; 3. Abstract hardware peripheral manager module; 4. Hardware peripheral call interface object module; 5. Remote call interface module; 6. Remote call processor module; 7. Plugin extension module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0018] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0019] Please refer to Figure 1 and Figure 2For example, if the original remote video assistance and remote device call cannot run the local business system in the remote terminal due to various restrictions, it is necessary to develop an agent system to call the hardware peripherals and collect the corresponding data, and send it to the local business system through transfer. The embodiment of the present application discloses a video assistance and device remote call method based on WebRTC, including: When the WebRTC connection is ready, the local terminal initiates a request to obtain device information to the remote terminal. After receiving the request, the remote terminal collects the device information through the hardware peripheral manager and returns it to the remote terminal. After receiving the device information, the local terminal performs an interface compatibility check on the hardware peripheral call interface information contained therein. If the interface is incompatible, the user is prompted to perform a version matching upgrade, otherwise proceed to the next step; After the local terminal completes the device information check and verification, it checks whether the corresponding version of the remote hardware peripheral call plug-in has been installed locally; if the corresponding version of the plug-in has been installed, it proceeds to the step of creating a remote hardware peripheral object, otherwise it proceeds to the next step; The local terminal sends a request to obtain a plug-in for a remote hardware peripheral to the remote terminal; after receiving the request, the remote terminal packages the plug-in based on HTML+CSS+JavaScript and sends it to the local terminal through the WebRTC data channel; After receiving the plug-in, the local terminal verifies, unpacks, installs, loads and registers the plug-in, then creates the corresponding remote hardware peripheral object and hands it over to the abstract hardware peripheral manager for management and scheduling; When the business system initiates a device call request, and the user directs the request to a remote hardware peripheral object through selection or configuration, the remote hardware peripheral object is responsible for encapsulating the call request and the corresponding related parameters and sending them to the remote terminal; After receiving the call request, the remote terminal transfers the request flow to the remote call processor for analysis, and then assigns it to the hardware peripheral manager. The hardware peripheral manager schedules and drives the corresponding hardware peripherals to collect data, and finally the remote call processor transmits the collected data to the local terminal; After the local terminal receives the collected data, it processes the collected data and finally submits it to the business system.

[0020] Compared with the remote desktop method, the technical solution provided by the present invention can directly call the hardware peripherals of the remote terminal in the local terminal in a remote calling manner, collect corresponding data and enter it directly into the business system in the local terminal, without the need to open and run the business system in the remote terminal.

[0021] Compared with the remote terminal proxy service program, the technical solution provided by the present invention can call the hardware peripherals of the remote terminal more flexibly, and at the same time can well support the device types that output multimedia stream data.

[0022] Hardware peripherals can be divided into two categories according to the way of outputting data - one is block devices that output text, images, files, etc.; the other is stream devices that output multimedia audio and video, etc. The technical solution of the present invention is mainly based on the WebRTC technology for network connection and communication. Among them, the multimedia channel of WebRTC can be used to transmit the output data of stream devices, and the data channel can be used to transmit the output data of block devices.

[0023] In the technical solution of the present invention, the remote hardware peripheral object of the local terminal comes from the plug-in provided by the remote terminal. The remote terminal packs the remote hardware peripheral plug-in based on HTML+CSS+JavaScript according to the specification of the browser container plug-in extension mechanism and transmits it to the local terminal through the data channel of WebRTC. The browser container of the local terminal unpacks, verifies, installs and registers it, and finally hands it over to the abstract hardware peripheral manager for management. Among them, the remote hardware peripheral object is responsible for encapsulating the remote hardware peripheral call request and related parameters of the local terminal business system, sending them to the remote terminal through the data channel of WebRTC. The remote call processor in the browser container of the remote terminal is responsible for parsing the request and allocating it to the hardware peripheral manager of the remote terminal, and finally transmitting the collected data back to the local terminal.

[0024] A device is applied to a video assistance and device remote call method based on WebRTC as described above, and is characterized in that it includes: a browser container device end and a WebRTC technology end, and the browser container device end includes: Hardware Peripheral Call Module 1: Hardware Peripheral Call Module 1 can drive the hardware peripherals on the terminal, initiate calls and collect corresponding data; Hardware Peripheral Manager Module 2: Hardware Peripheral Manager Module 2 can manage various hardware peripherals integrated in the terminal, and uniformly allocate and schedule them; Abstract Hardware Peripheral Manager Module 3: The browser container device end has the ability to abstractly manage hardware peripherals. In addition to managing the hardware peripherals on the local terminal, it can also abstract and manage the hardware peripherals on the remote terminal, and uniformly allocate and schedule them; Hardware Peripheral Call Interface Object Module 4: The browser container device end provides corresponding unified hardware peripheral interface objects for the business system according to the type of hardware peripherals. The business system can drive the hardware peripherals and collect corresponding data by calling the interface objects; Remote Call Interface Module 5: The remote call interface abstracts the remote call method and parameters to form a unified call interface, enabling different network communication methods to be called in a unified manner; Remote Call Processor Module 6: The remote call manager on the browser container device side is mainly responsible for parsing, allocating, and responding to remote call requests; Plugin Extension Module 7: The browser container provides a plugin extension mechanism based on HTML+CSS+JavaScript technology, supporting functions such as plugin verification, installation, loading, registration, management, and uninstallation.

[0025] The multimedia channel on the WebRTC technology side can be used to transmit the output data of streaming devices, while the data channel can be used to transmit the output data of block devices.

[0026] After receiving a call request, the Hardware Peripheral Call Interface Object Module 4 encapsulates the call request and corresponding related parameters, and then sends them to the Abstract Hardware Peripheral Manager. The Hardware Peripheral Abstract Manager schedules the hardware peripheral objects according to user selection or configuration.

[0027] The Remote Call Processor Module 6 receives and parses the hardware peripheral remote call request from the peer end, then allocates the request to the Abstract Hardware Peripheral Manager. The Abstract Hardware Peripheral Manager is responsible for driving the call to the local hardware peripheral and collecting the corresponding data. Finally, the Remote Call Processor encapsulates the result data and responds and returns it to the peer end.

[0028] The Plugin Extension Module 7 depends on this plugin extension mechanism, dynamically installs and registers remote hardware peripheral plugins, creates remote hardware peripheral objects, and hands them over to the Abstract Hardware Peripheral Manager for management.

[0029] It is worth mentioning that the above solution enables the support of remote calls for block devices and streaming devices based on WebRTC technology, and based on the overall architecture design solution of the browser container, it can realize the dynamic installation and creation of plugin and hardware peripheral remote call objects.

[0030] In the above embodiments, the device components involved are all conventional device components unless otherwise specified, and the connection methods and control methods involved are all conventional connection methods and control methods unless otherwise specified.

[0031] The present invention has been described in detail above in conjunction with the embodiments. However, those skilled in the art can understand that without departing from the purpose of the present invention, various specific parameters in the above embodiments can be changed to form multiple specific embodiments, which are all within the common variation range of the present invention and will not be elaborated here one by one.

Claims

1. A video assistance and device remote calling method based on WebRTC, characterized in that: include: After the WebRTC connection is established, the local terminal initiates a request to obtain device information from the remote terminal; After receiving the request for obtaining device information, the remote terminal collects the device information through the hardware peripheral manager and returns it to the remote terminal; After receiving the device information, the local terminal performs an interface compatibility check on the hardware peripheral call interface information contained therein. If the interface is incompatible, the user is prompted to perform a version matching upgrade, otherwise proceed to the next step; After the local terminal completes the device information check and verification, it checks whether the corresponding version of the remote hardware peripheral call plug-in has been installed locally; If the corresponding version of the plug-in has been installed, proceed to the step of creating a remote hardware peripheral object, otherwise proceed to the next step; The local terminal sends a request to obtain a plug-in for a remote hardware peripheral to the remote terminal; after receiving the request, the remote terminal packages the plug-in based on HTML+CSS+JavaScript and sends it to the local terminal through the WebRTC data channel; After receiving the plug-in, the local terminal verifies, unpacks, installs, loads and registers the plug-in, then creates the corresponding remote hardware peripheral object and hands it over to the abstract hardware peripheral manager for management and scheduling; When the business system initiates a device call request, and the user directs the request to a remote hardware peripheral object through selection or configuration, the remote hardware peripheral object is responsible for encapsulating the call request and the corresponding related parameters and sending them to the remote terminal; After receiving the call request, the remote terminal transfers the request flow to the remote call processor for analysis, and then assigns it to the hardware peripheral manager. The hardware peripheral manager schedules and drives the corresponding hardware peripherals to collect data, and finally the remote call processor transmits the collected data to the local terminal; After the local terminal receives the collected data, it processes the collected data and finally submits it to the business system.

2. A device, applied to a video assistance and device remote calling method based on WebRTC as claimed in claim 1, characterized in that: include: The browser container device end and the WebRTC technology end, the browser container device end includes: Hardware peripheral calling module: The hardware peripheral calling module can drive the hardware peripherals on the terminal, initiate calls and collect corresponding data; Hardware peripheral manager module: The hardware peripheral manager module can manage various hardware peripherals integrated in the terminal, and uniformly allocate and schedule them; Abstract hardware peripheral manager module: The browser container device has the ability to abstractly manage hardware peripherals. In addition to managing hardware peripherals on the local terminal, it can also abstract and manage hardware peripherals on remote terminals, and uniformly allocate and schedule them. Hardware peripheral calling interface object module: The browser container device provides the business system with a corresponding unified hardware peripheral interface object according to the type of hardware peripherals. The business system can drive the hardware peripherals and collect corresponding data by calling the interface object; Remote call interface module: The remote call interface abstracts the remote call method and parameters to form a unified call interface, so that different network communication methods can be called in a unified way; Remote call processor module: The remote call manager on the browser container device is mainly responsible for parsing, allocating and responding to remote call requests; Plug-in extension module: The browser container provides a plug-in extension mechanism based on HTML+CSS+JavaScript technology, supporting plug-in verification, installation, loading, registration, management and uninstallation functions.

3. A device according to claim 2, characterized in that: The multimedia channel of the WebRTC technology end can be used to transmit the output data of the streaming device, while the data channel can be used to transmit the output data of the block device.

4. A device according to claim 2, characterized in that: After receiving the call request, the hardware peripheral call interface object module will encapsulate the call request and corresponding related parameters, and then send them to the abstract hardware peripheral manager, which will schedule the hardware peripheral object according to the user's selection or configuration.

5. A device according to claim 2, characterized in that: The remote call processor module receives and parses the remote call request of the hardware peripheral of the other end, and then assigns the request to the abstract hardware peripheral manager, which is responsible for driving the call of the local hardware peripheral and collecting the corresponding data. Finally, the remote call processor encapsulates the result data and returns the response to the other end.

6. A device according to claim 2, characterized in that: The plug-in extension module relies on the plug-in extension mechanism to dynamically install and register remote hardware peripheral plug-ins, create remote hardware peripheral objects, and hand them over to the abstract hardware peripheral manager for management.

Images