A device and method for recognizing a two-dimensional code, binding, and interacting with a video

Abstract

The application discloses a two-dimensional code recognition binding and video interaction device and method, which comprises the following steps: a user front end carries out code recognition through a web page browser supporting webRTC, and associates and binds an audio and video equipment terminal and the user front end; the user front end initiates audio and video interaction through the web page browser supporting webRTC; a streaming media distribution module based on GB28181 / ONVIF protocol outputs an audio and video stream to the audio and video equipment terminal, and based on the webRTC protocol, outputs the audio and video stream to the user front end.

Description

Technical Field

[0001] This invention pertains to QR code-based interactive technology, specifically a device and method for QR code recognition, binding, and video interaction. Background Technology

[0002] like Figure 1 As shown, the existing video call function of video surveillance mainly consists of the management terminal of the monitoring system, the monitoring system software, and matching microphones and speakers with sound pickup capabilities. The existing video call operation of video surveillance is achieved by connecting a remote video surveillance camera with sound pickup and playback capabilities through the monitoring system software and microphone on the management terminal of the monitoring system. This method only has the function of the system management terminal to initiate the conversation, while the remote camera is the called end. Since the operation must be carried out through limited software and network, the existing conventional video call method of video surveillance is inconvenient. Therefore, the existing video call function of video surveillance has the following disadvantages: (1) Most only support voice calls, and the operation of establishing a call is mostly carried out on the software of the service management terminal; (2) Most do not support cross-system, cross-network, and cross-platform calls; (3) The user operation is relatively inconvenient. Users need to operate in a specific software and hardware environment to establish a call, and users need to have certain professional knowledge.

[0003] like Figure 2 As shown, existing video surveillance manufacturers all provide the function of remotely monitoring and communicating with smart cameras after scanning a QR code with a smart terminal (mobile phone) APP. The smart camera is numbered using a private protocol, and this device number and server link address are affixed to the smart camera via a QR code when the device is manufactured. After the smart camera is purchased by the user and successfully connected to the network, it will register and log in to the manufacturer's cloud video monitoring and communication system through the private protocol. At this time, the smart camera is online and unbound in the manufacturer's cloud video monitoring and communication system. The user scans the QR code with the manufacturer's APP and binds the device to the user's APP account. At this time, the user can access the smart camera through the manufacturer's APP to remotely monitor or communicate with it. However, the existing video surveillance manufacturers' function of remotely monitoring and communicating with smart cameras after scanning a QR code with a mobile phone APP has the following disadvantages: (1) It does not support cross-platform and cross-network, and can only be implemented within the application software and private network of some video surveillance manufacturers; (2) The operation of the user is still not simple enough, and there is still difficulty in operation for users who lack professional knowledge. Summary of the Invention

[0004] To overcome the shortcomings and problems of existing technologies, this invention provides a method for QR code recognition, binding, and video interaction.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A device for QR code recognition binding and video interaction includes a user front-end with communication connection, a multi-protocol WebRTC audio and video communication system, and an audio and video device terminal. The multi-protocol WebRTC audio and video communication system includes a user management system, a communication scheduling system, and a signaling and session system with communication connection. The signaling and session system includes a multi-protocol SIP signaling module, a streaming module, a streaming merging module, and a streaming media distribution module with communication connection. The audio and video device terminal includes a device QR code.

[0007] Preferably, the user front-end is used to scan and identify the device's QR code using a web browser that supports WebRTC, parse the device's QR code to obtain an access link and the audio / video device terminal code, parse the access link to obtain a registration / login page opened through a web browser that supports WebRTC, obtain a registration / login request through the registration / login page, verify the registration / login request, and if the registration / login request is verified, a binding request is obtained. Based on the binding request, it is confirmed whether to bind the audio / video device terminal. If so, a confirmation binding request is generated. Based on the confirmation binding request, a query request, the audio / video device terminal code, and a user code are generated. Based on the query request, the audio / video device terminal code, and the user code, the audio / video device terminal code corresponding to the audio / video device terminal code is... The video device terminal and the user front-end corresponding to the user code are associated and bound. A page call request is initiated through a web browser that supports WebRTC. The audio and video session resource request is obtained based on the page call request. After the audio and video session resource request, the scheduling pull service resource instruction and the allocation merge service resource instruction are obtained. The pull module and the merge module are started according to the scheduling pull service resource instruction and the allocation merge service resource instruction. The pull module retrieves the user audio and video stream based on the WebRTC protocol and the terminal audio and video stream based on the GB28181 / ONVIF protocol. The merge module combines the user audio and video stream and the terminal audio and video stream according to a fixed screen layout and plays the audio and video merge based on the WebRTC protocol.

[0008] Preferably, the multi-protocol WebRTC audio and video communication system is used to associate and bind the audio and video device terminal corresponding to the audio and video device terminal encoding and the user front-end corresponding to the user encoding.

[0009] Preferably, the user management system is used to parse the device QR code to obtain the access link and the audio / video device terminal code, open the registration / login page according to the access link, obtain the registration / login request through the registration / login page, verify the registration / login request, and if the registration / login request is verified, obtain the binding request, confirm whether to bind the audio / video device terminal according to the binding request, generate a confirmation binding request according to the confirmation binding request, generate a query request according to the confirmation binding request, and send the query request, the audio / video device terminal code, and the user code to the multi-protocol WebRTC audio / video communication system.

[0010] Preferably, the signaling and session system is used to obtain audio and video session resource requests based on page call requests, obtain scheduling pull service resource instructions and allocation merging service resource instructions based on audio and video session resource requests, and start the pull module and merging module based on the scheduling pull service resource instructions and allocation merging service resource instructions.

[0011] Preferably, the streaming module is used to send a front-end streaming request and a terminal streaming request to the user front-end and the audio / video device terminal, respectively; the merging module is used to acquire the user audio / video stream and the terminal audio / video stream and merge them into an audio / video merging stream according to a fixed screen layout; and the streaming media distribution module is used to acquire the audio / video merging stream and output the audio / video merging stream to the audio / video device terminal based on the GB28181 / ONVIF protocol and to output the audio / video merging stream to the user front-end based on the WebRTC protocol.

[0012] Preferably, the audio and video device terminal is used to acquire terminal audio and video streams based on the GB28181 / ONVIF protocol, synthesize audio and video streams according to the user audio and video streams and terminal audio and video streams and according to a fixed screen layout, and play the audio and video streams based on the GB28181 / ONVIF protocol.

[0013] A method for QR code recognition, binding, and video interaction, employing the aforementioned apparatus for QR code recognition, binding, and video interaction, includes the following steps:

[0014] Step 1: The user's front end scans the device's QR code using a web browser that supports WebRTC, and sends the device's QR code and user code to the user management system;

[0015] Step 2: The user management system parses the device QR code to obtain the access link and audio / video device terminal code, and sends the access link back to the user's front end;

[0016] Step 3: The user's front-end parses the access link to obtain the registration / login page, which is opened through a web browser that supports WebRTC. The user's front-end obtains the registration / login request through the registration / login page and sends it to the user management system.

[0017] Step 4: The user management system verifies the registration / login request. If the user management system verifies the registration / login request successfully, a binding request is sent back to the user's front end.

[0018] Step 5: The user front-end confirms whether to bind the audio / video device terminal. If so, a confirmation binding request is generated and sent to the user management system.

[0019] Step 6: When the user management system receives the confirmation binding request, it generates a query request and sends the query request, audio and video device terminal code, and user code to the multi-protocol WebRTC audio and video communication system;

[0020] Step 7: The multi-protocol WebRTC audio and video communication system associates and binds the audio and video device terminal corresponding to the audio and video device terminal encoding with the user front-end corresponding to the user encoding;

[0021] Step 8: The user's front end sends a page call request to the signaling and session system through a web browser that supports WebRTC;

[0022] Step 9: After receiving the page call request, the signaling and session system obtains the audio and video session resource request and sends it to the signaling and session system;

[0023] Step 10: After receiving the audio and video session resource request, the signaling and session system obtains the instruction to schedule streaming service resources and the instruction to allocate merging service resources;

[0024] Step 11: The signaling and session system starts the streaming module and the merging module according to the scheduling streaming service resource instruction and the allocation merging service resource instruction. The streaming module sends a front-end streaming request and a terminal streaming request to the user front-end and the audio / video device terminal, respectively. After receiving the front-end streaming request, the user front-end sends the user audio and video stream based on the WebRTC protocol to the streaming module. After receiving the terminal streaming request, the audio / video device terminal sends the terminal audio and video stream based on the GB28181 / ONVIF protocol to the streaming module. The merging module obtains the user audio and video stream and the terminal audio and video stream and combines them into a combined audio and video stream according to a fixed screen layout. The streaming media distribution module obtains the combined audio and video stream and outputs the combined audio and video stream to the audio / video device terminal based on the GB28181 / ONVIF protocol, and outputs the combined audio and video stream to the user front-end based on the WebRTC protocol.

[0025] Preferably, step 7 specifically includes:

[0026] When a multi-protocol WebRTC audio and video communication system receives a query request, it determines whether the audio and video device terminal is online. If so, it associates and binds the audio and video terminal corresponding to the audio and video terminal encoding with the user front-end corresponding to the user encoding; otherwise, it initializes the system.

[0027] Preferably, the audio / video device terminal periodically sends a heartbeat cycle to the multi-protocol WebRTC audio / video communication system, and the multi-protocol WebRTC audio / video communication system determines whether the audio / video device terminal is online based on the heartbeat cycle.

[0028] The outstanding and beneficial technical effects of this invention compared to the prior art are:

[0029] In this invention, a web browser that supports WebRTC is used to quickly scan and register / bind the device, without the need to download an application app in advance. Moreover, the WebRTC audio and video communication system based on multiple protocols enables automatic registration and connection of audio and video device terminals. Therefore, this QR code recognition, binding and video interaction method has the advantages of being ready to use immediately, highly automated, reducing the user's usage threshold, improving user efficiency and being easy to operate.

[0030] In this invention, a web browser supporting WebRTC and an audio / video device terminal supporting the GB28181 / ONVIF protocol are used to achieve audio / video interaction. Since WebRTC itself is just a client library, no installation is required on the user's front end. Quick registration and binding can be achieved directly by scanning a QR code with a web browser, unlike existing technologies that require downloading a dedicated app for registration and binding. This enables audio / video interaction between a web browser with WebRTC as the main communication interaction subject and an audio / video device terminal based on the GB28181 / ONVIF protocol, achieving cross-platform and cross-terminal audio / video interaction. Therefore, this QR code recognition binding and video interaction method has the advantages of being ready to use immediately, highly automated, lowering the user's threshold, improving user efficiency, and being sufficiently simple to operate. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the steps involved in a video call during standard video surveillance.

[0032] Figure 2 This is a flowchart illustrating the steps for existing video surveillance manufacturers to scan a QR code to view video surveillance footage and make remote calls.

[0033] Figure 3 This is a schematic diagram of the frame of the device of the present invention;

[0034] Figure 4 This is a schematic diagram of the steps of the present invention;

[0035] Figure 5 This is a schematic diagram of the swimlane timing during QR code binding in this invention;

[0036] Figure 6 This is a schematic diagram of the swimlane timing during audio and video interaction in this invention;

[0037] Figure 7 This is a schematic diagram of the swimlane timing at the end of a call in this invention. Detailed Implementation

[0038] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0039] This embodiment provides a device for QR code recognition and binding and video interaction, including a user front-end with communication connection, a multi-protocol WebRTC audio and video communication system, and an audio and video device terminal. The multi-protocol WebRTC audio and video communication system includes a user management system, a communication scheduling system, and a signaling and session system with communication connection. The signaling and session system includes a multi-protocol SIP signaling module, a streaming module, a merging module, and a streaming media distribution module with communication connection. The audio and video device terminal includes a device QR code.

[0040] The user front-end is used to scan and identify the device's QR code using a web browser that supports WebRTC, parse the QR code to obtain an access link and the audio / video device terminal code, parse the access link to open a registration / login page through the web browser that supports WebRTC, obtain a registration / login request through the registration / login page, verify the registration / login request, and if the registration / login request is verified, a binding request is obtained. Based on the binding request, it is confirmed whether to bind the audio / video device terminal. If so, a confirmation binding request is generated. Based on the confirmation binding request, a query request, the audio / video device terminal code, and a user code are generated. Based on the query request, the audio / video device terminal code, and the user code, the corresponding audio / video device is connected. The backup terminal and the user front-end corresponding to the user code are associated and bound. A page call request is initiated through a web browser that supports WebRTC. The audio and video session resource request is obtained based on the page call request. After the audio and video session resource request, the scheduling pull service resource instruction and the allocation merge service resource instruction are obtained. The pull module and the merge module are started according to the scheduling pull service resource instruction and the allocation merge service resource instruction. The pull module retrieves the user audio and video stream based on the WebRTC protocol and the terminal audio and video stream based on the GB28181 / ONVIF protocol. The merge module combines the user audio and video stream and the terminal audio and video stream according to a fixed screen layout and plays the audio and video merge based on the WebRTC protocol.

[0041] The multi-protocol WebRTC audio and video communication system is used to associate and bind the audio and video device terminal corresponding to the audio and video device terminal encoding and the user front-end corresponding to the user encoding.

[0042] The user management system is used to parse the device QR code to obtain the access link and audio / video device terminal code, open the registration / login page according to the access link, obtain the registration / login request through the registration / login page, verify the registration / login request, and if the registration / login request is verified, a binding request is obtained. The binding request is used to confirm whether to bind the audio / video device terminal. If so, a confirmation binding request is generated. A query request is generated based on the confirmation binding request, and the query request, audio / video device terminal code, and user code are sent to the multi-protocol WebRTC audio / video communication system.

[0043] The signaling and session system is used to obtain audio and video session resource requests based on page call requests, obtain scheduling pull service resource instructions and allocation merge service resource instructions based on audio and video session resource requests, and start the pull module and merge module based on the scheduling pull service resource instructions and allocation merge service resource instructions.

[0044] The streaming module is used to send streaming requests to the user front-end and the audio / video device terminal respectively. The merging module is used to acquire the user's audio / video stream and the terminal's audio / video stream and merge them into a single audio / video stream according to a fixed screen layout. The streaming media distribution module is used to acquire the audio / video merged stream and output it to the audio / video device terminal based on the GB28181 / ONVIF protocol and to the user front-end based on the WebRTC protocol.

[0045] The audio and video device terminal is used to acquire terminal audio and video streams based on the GB28181 / ONVIF protocol, synthesize audio and video streams according to the user audio and video streams and terminal audio and video streams in a fixed screen layout, and play the audio and video streams based on the GB28181 / ONVIF protocol.

[0046] It should be noted that the QR code recognition binding and video interaction method provided in this embodiment enables QR code scanning binding and audio / video interaction between the user's front-end and the audio / video device terminal via a multi-protocol WebRTC audio / video communication system. The user's front-end, the WebRTC audio / video communication system, and the audio / video device terminal can communicate and interact with each other via the Internet. The user management system, communication scheduling system, and signaling and session system can also communicate and interact with each other via the Internet. The multi-protocol SIP signaling module, streaming module, merging module, and streaming media distribution module can also communicate and interact with each other via the Internet.

[0047] A user front-end is a smart terminal equipped with a web browser supporting WebRTC, QR code scanning and recognition, and audio / video calling capabilities. The user front-end uses this QR code recognition and video interaction method to identify and bind audio / video device terminals and perform audio / video interaction. The user front-end can be a mobile phone. A web browser supporting WebRTC can be opened on the user front-end. On this web browser, the user can enter a registration / login request, thereby completing user registration / login on a multi-protocol WebRTC audio / video communication system. The user can also confirm whether to bind an audio / video device terminal, thus associating the corresponding audio / video device terminal with the user front-end corresponding to the user code through the multi-protocol WebRTC audio / video communication system. Finally, the user can decide whether to send a page call request, thereby enabling audio / video interaction between the associated audio / video device terminal and the user front-end.

[0048] WebRTC (Web Real-Time Communication) is used to enable browser-based audio and video calls. WebRTC is a client library that requires no installation on the user's front-end; audio and video calls can be made simply by using the user's pre-installed browser. Existing browsers on Android and iOS devices already support WebRTC. For example, iOS browsers support VP9, ​​and Firefox has added retransmission and Transport-wide Congestion Control. In addition to supporting WebRTC, it also provides 4K, HDR images, and surround sound, improving the user's audio and video call experience.

[0049] The multi-protocol WebRTC audio and video communication system is a communication system capable of real-time audio and video interaction, supporting both the WebRTC protocol and the GB28181 / ONVIF protocol. It includes a user management system, a communication scheduling system, and a signaling and session system. The signaling and session system comprises a multi-protocol SIP signaling module, a streaming module, a streaming merging module, and a streaming media distribution module. Through the multi-protocol WebRTC audio and video communication system, interactive services between user front-ends and device terminals are realized.

[0050] ONVIF is one of the international standard protocols for network cameras, and GB / T28181 is the national standard for network cameras. ONVIF (Open Network Video Interface Forum) develops open industry standards based on the principles of openness and transparency. The interfaces defined in the device management and control sections of the ONVIF specification are provided in the form of Web Services. The ONVIF specification covers complete XML and WSDL definitions. Every terminal device supporting the ONVIF specification must provide a Web Service corresponding to its functionality. Data interaction between the server and client uses the SOAP protocol. Other parts of ONVIF, such as audio and video streams, are handled via RTP / RTSP. On June 1, 2012, the Ministry of Public Security issued the "Technical Requirements for Information Transmission, Exchange, and Control of Security Video Surveillance Network Systems" (GB / T28181), officially putting the monitoring network technology standard into effect. The GB28181 transmission protocol is responsible for converting the PS stream pushed by GB28181 devices / platforms into ES streams, and then distributing them in multiple formats such as RTSP, RTMP, FLV, and HLS, enabling plugin-free playback on various terminals such as web browsers, mobile browsers, WeChat, and PC clients. The GB28181 / ONVIF protocol is currently the standard platform access protocol for mainstream network cameras, and all support audio and video interaction using the RTSP streaming media protocol.

[0051] The audio / video device terminal includes a device QR code, which contains an access link and information such as the audio / video device terminal encoding based on the GB28181 / ONVIF protocol. This GB28181 / ONVIF protocol-based encoding is used to request audio / video interaction from a multi-protocol WebRTC audio / video communication system. This audio / video device terminal supports the GB28181 / ONVIF protocol, audio input / output, and RTSP / HTTP protocols. It connects to the user's front end via a GB28181 / ONVIF protocol-based and multi-protocol WebRTC audio / video communication system. The device QR code can be affixed to the outer surface of the audio / video device terminal, which can be a network camera (IPC).

[0052] On the other hand, this embodiment also provides a method for QR code recognition, binding, and video interaction, which uses the above-mentioned device for QR code recognition, binding, and video interaction, and includes the following steps:

[0053] Step 1: The user's front end scans the device's QR code using a web browser that supports WebRTC, and sends the device's QR code and user code to the user management system;

[0054] In the above steps, the user obtains the device's QR code by scanning it with a web browser that supports WebRTC. The device QR code contains audio / video device terminal encoding based on the GB28181 / ONVIF protocol. With a normal internet connection, the user's front-end sends the device QR code and user encoding to the user management system. The user encoding can be pre-set in the user's front-end. Figure 4 The diagram shows a flowchart of the present invention. In the diagram, S4 represents the user scanning the device QR code using a web browser that supports WebRTC, and S5 represents the user sending the device QR code and user code to the user management system from the user's front end.

[0055] Step 2: The user management system parses the device QR code to obtain the access link and audio / video device terminal code, and sends the access link back to the user's front end;

[0056] In the above steps, the access link is used to register / login user accounts, and the audio / video device terminal code is used to query the specified audio / video device terminal.

[0057] Step 3: The user's front-end parses the access link to obtain the registration / login page, which is opened through a web browser that supports WebRTC. The user's front-end obtains the registration / login request through the registration / login page and sends it to the user management system.

[0058] In the steps described above, a web browser that supports WebRTC opens a registration / login page, where users can complete their registration / login requests. The registration / login request includes a user account to save the user's personal information, habits, and viewing history. Figure 4 S6 in the diagram represents the user's front-end parsing the access link to obtain the registration / login page, which is then opened by a web browser that supports WebRTC. S7 represents the user's front-end sending the registration / login request to the user management system.

[0059] Step 4: The user management system verifies the registration / login request. If the user management system verifies the registration / login request successfully, a binding request is sent back to the user's front end.

[0060] In the above steps, the registration / login request includes a user account, and the user management system verifies the registration / login request by verifying the user account. During the user account verification process, the user management system has pre-stored accounts. The user management system determines whether the registration / login account is a pre-stored account. If so, it retrieves the account information corresponding to the pre-stored account and sends it to the user's front end, and also sends the current device status information of the audio / video device terminal to the user's front end; otherwise, it creates a new account for the registration / login account and stores it in the user management system, and sends the newly created account information to the user's front end, as well as the current device status information of the audio / video device terminal to the user's front end.

[0061] Step 5: The user front-end confirms whether to bind the audio / video device terminal. If so, a confirmation binding request is generated and sent to the user management system.

[0062] In the above steps, the binding request is used to allow the user to confirm whether to perform the association binding on the user's front end. The binding request can be visually displayed on the user's browser, for example, as a pop-up window. The user can select whether to perform the association binding by clicking on the request in the user's browser. If the user selects to perform the association binding, a binding request is sent to the user management system. If the user refuses to perform the association binding, the WebRTC-based web browser QR code scanning and binding process ends.

[0063] Step 6: When the user management system receives the confirmation binding request, it generates a query request and sends the query request, audio and video device terminal code, and user code to the multi-protocol WebRTC audio and video communication system;

[0064] In the above steps, the query request is used to check whether the audio / video device terminal associated with the audio / video device terminal code is online. If the audio / video device terminal associated with the audio / video device terminal code is online, the multi-protocol WebRTC audio / video communication system will associate and bind the audio / video device terminal corresponding to the audio / video device terminal code with the user front-end corresponding to the user code. Figure 4 S8 in the context refers to the operation where the user sends a query request, audio / video device terminal code, and user code to the communication scheduling system.

[0065] Step 7: The multi-protocol WebRTC audio and video communication system associates and binds the audio and video device terminal corresponding to the audio and video device terminal encoding with the user front-end corresponding to the user encoding, thus completing the QR code association and binding;

[0066] In the above steps, after the audio and video device terminal corresponding to the audio and video device terminal code and the user front-end corresponding to the user code are associated and bound, the application page is visually displayed on the browser of the user front-end. The application page visually displays a list of devices. Users can view the device information of the audio and video device terminal through the device list, and can initiate audio and video interactions with the audio and video device terminal through the browser of the user front-end.

[0067] Step 8: The user's front end sends a page call request to the signaling and session system through a web browser that supports WebRTC;

[0068] In the above steps, the user sends a page call request to the multi-protocol SIP signaling module of the signaling and session system through the user's browser. The page call request is used to initiate audio and video interaction with the multi-protocol SIP signaling module. Figure 4 S11 in the middle refers to the operation of maintaining audio and video sessions between the user front-end and the multi-protocol SIP signaling module.

[0069] Step 9: After receiving the page call request, the signaling and session system obtains the audio and video session resource request;

[0070] In the above steps, the multi-protocol SIP signaling module sends the audio and video session resource request to the streaming module. The audio and video session resource request is used to initiate the scheduling of audio and video streams to the streaming module, thereby starting the audio and video interaction between the user front-end and the audio and video device terminal. Figure 4 S12 in the text indicates the operation of the multi-protocol SIP signaling module sending audio and video session resource requests to the streaming module.

[0071] Step 10: The signaling and session system obtains the instruction to schedule streaming service resources and the instruction to allocate merging service resources based on the audio and video session resource request;

[0072] In the above steps, the command to schedule streaming service resources is used to start the streaming module, and the command to allocate streaming service resources is used to start the streaming module.

[0073] Step 11: The signaling and session system starts the streaming module and the merging module according to the scheduling streaming service resource instruction and the allocation merging service resource instruction. The streaming module sends the front-end streaming request and the terminal streaming request to the user front-end and the audio and video device terminal respectively. After receiving the front-end streaming request, the user front-end sends the user audio and video stream based on the WebRTC protocol to the streaming module. After receiving the terminal streaming request, the audio and video device terminal sends the terminal audio and video stream based on the GB28181 / ONVIF protocol to the streaming module. The merging module obtains the user audio and video stream and the terminal audio and video stream and synthesizes the audio and video merged stream according to a fixed screen layout. The streaming media distribution module obtains the audio and video merged stream and outputs the audio and video merged stream to the audio and video device terminal based on the GB28181 / ONVIF protocol, and outputs the audio and video merged stream to the user front-end based on the WebRTC protocol.

[0074] In the above steps, audio-video merging refers to the simultaneous playback of the user's audio-video stream and the terminal's audio-video stream. Synthesizing audio-video streams according to a fixed screen layout refers to the audio-video streams from the audio-video device terminal and the user's front-end being arranged in a certain proportion on the same application page. This fixed screen layout can be pre-set in the signaling and session system. Figure 4 S13 indicates the operation of the pull module to pull the user's audio and video streams in real time from the user's front-end via WebRTC for the front-end browser. S16 indicates the operation of the pull module establishing a session based on the GB28181 / ONVIF protocol with the audio and video device terminal and the pull module pulling the terminal's audio and video streams in real time. S14 indicates the operation of the pull module sending the pulled user audio and video streams and terminal audio and video streams to the merge module. At this time, the user audio and video streams and terminal audio and video streams in the same time period can have the same session tag to ensure audio synchronization. S15 indicates the operation of the merge module sending the merged audio and video streams to the streaming media distribution module. S17 indicates the operation of the streaming media distribution module pushing the merged audio and video streams based on the WebRTC protocol with session tags to the user's front-end. S18 indicates the operation of the streaming media distribution module pushing the merged audio and video streams based on the GB28181 / ONVIF protocol with session tags to the audio and video terminal device, thereby realizing audio and video interaction.

[0075] Step 7 specifically includes:

[0076] When a multi-protocol WebRTC audio and video communication system receives a query request, it determines whether the audio and video device terminal is online. If it is, it associates and binds the audio and video terminal corresponding to the audio and video terminal encoding with the user front-end corresponding to the user encoding. Otherwise, it initializes the system.

[0077] In the above steps, the audio / video device terminal has two states: online and offline. If the associated audio / video device terminal is offline, the multi-protocol WebRTC audio / video communication system sends an offline notification to the user frontend, initializes the system, and then terminates the current process. The multi-protocol WebRTC audio / video communication system can monitor the online status of the audio / video device terminal in real time. Figure 4 As shown, S9 represents the operation of the communication scheduling system to send an offline notification to the user's front end after determining that the audio / video device terminal is offline, and S10 represents the operation of the communication scheduling system to query the status of the audio / video device terminal from the multi-protocol SIP signaling module according to the query request.

[0078] The audio and video device terminal periodically sends a heartbeat cycle to the multi-protocol WebRTC audio and video communication system, which determines whether the audio and video device terminal is online based on the heartbeat cycle.

[0079] like Figure 5 The diagram shown is a swimlane timing diagram for QR code binding in this invention. If the audio / video device terminal is online, it sends a connection request via the GB28181 / ONVIF protocol (i.e., ...). Figure 5 (20) Automatic connection to the signaling and session system, which periodically records the audio and video device terminal encoding under the GB28181 / ONVIF protocol (i.e. Figure 5 21) and the online status of audio and video equipment terminals (i.e. Figure 5 (22) The audio and video equipment terminal periodically transmits signals through a heartbeat cycle (i.e., Figure 5 23) Report its online status to the multi-protocol WebRTC audio and video communication system.

[0080] If the user's front end uses a browser that supports WebRTC to scan the QR code affixed to the surface of the audio / video terminal device (i.e. Figure 5 In step 24), the device QR code contains an access link and information such as the audio / video device terminal encoding based on the GB28181 / ONVIF protocol (i.e., Figure 5 If step 25 in the code is executed, the user's browser will open the user registration / login page (i.e., ...). Figure 5 (26) Then users can register / log in on this page (i.e. Figure 5 (27) After a user completes the registration or login process, the user's front end sends a registration / login request to the user management system (i.e., ...). Figure 5 28), used to manage the system to complete registration / login processing (i.e. Figure 5 (29) and send a notification to the browser indicating successful registration / login and redirecting the user to the application's homepage (i.e. Figure 5 In Figure 30), the application homepage displays a pop-up window prompting whether to bind a device (i.e., Figure 31). After the user clicks "Confirm Device Binding" in the pop-up window, the user's front-end sends a request to the user management system to bind the audio / video device terminal (i.e., ...). Figure 5 (32) After receiving the request, the user management system will query the signaling and session system to check whether the audio and video terminal equipment is online (i.e., Figure 5 (33) In this context, the signaling and session system processes query requests (i.e. Figure 5 (34) and feed back the query results to the user management system (i.e. Figure 5 (35) The user management system updates the audio and video device terminal status information on the application page (i.e. Figure 5 (36) and by prompting the user to confirm the binding in a pop-up window on the application page ( Figure 5 37) prompts the user to confirm the binding. After the user confirms, the binding confirmation request will be sent (i.e. Figure 5 38) is sent to the user management system. The user management system associates and binds the audio and video device terminal corresponding to the audio and video device terminal encoding with the user front-end corresponding to the user encoding (i.e., 39 in the figure), thereby completing the QR code scanning and recognition binding.

[0081] After the user's front-end and audio / video device terminal are associated and bound, the user can see the corresponding audio / video device terminal in the device list on the application page, and can view device information such as the previous status of the audio / video device terminal. At this time, the user can initiate an audio / video call from the device list, which includes two stages: establishing a call and ending a call.

[0082] like Figure 6 The diagram shown illustrates the swimlane timing diagram for establishing a call during audio-visual interaction in this invention. During the call establishment process, the user initiates a page call request to the signaling and session system through a WebRTC-enabled browser on their front end (i.e.,...). Figure 6 (40) After receiving the page call request, the signaling and session system requests audio and video session resources (i.e., Figure 6 41) Requests resources from the communication scheduling system. The communication scheduling system allocates pull service resources and merge service resources (i.e., ... Figure 6 (42) and will schedule the pull service resources and allocate the merge service resources (i.e. Figure 6 43) is sent to the signaling and session system. Upon receiving the instructions to schedule streaming service resources and allocate merging service resources, the signaling and session system immediately starts the streaming module and the merging module, and immediately streams to the audio and video equipment terminal via the GB28181 / ONVIF protocol (i.e., Figure 6 44) and pulling streams to the user's front end via the WebRTC protocol (i.e. Figure 6(45) At this point, the user front-end, upon receiving the pull request, transmits the stream to the signaling and session system via the GB28181 / ONVIF protocol (i.e., Figure 6 (46) The audio / video device terminal, upon receiving a pull request, transmits the stream to the signaling and session system via the WebRTC protocol (i.e., Figure 6 (47) After receiving the user's audio and video stream and the terminal's audio and video stream, the signaling and session system performs audio and video merging encoding (i.e., ... Figure 6 48) The audio and video streams are merged, which combines the user's audio and video streams and the terminal's audio and video streams according to a fixed screen layout. The merged audio and video streams are then transmitted to the audio and video device terminal via the GB28181 / ONVIF protocol. The merged audio and video streams are then transmitted to the user's front end via the WebRTC protocol, thereby completing the audio and video call between the user's front end and the audio and video device terminal.

[0083] like Figure 7 The diagram shown illustrates the swimlane timing diagram for the end of a call in this invention. During a normal audio / video call, if the user needs to end the call, they can initiate an end-of-call request on their front end or the audio / video device terminal (i.e.,...). Figure 7 (51) and through the signaling and session system, the session dissolution request (i.e. Figure 7 52) is sent to the communication scheduling system, which will then reclaim the pull service resources and allocate the merge service resources (i.e. Figure 7 53) is fed back to the signaling and session system, which then sends a session termination request under the WebRTC protocol to the audio / video device terminal (i.e., Figure 7 (54) also sends a session termination request under the GB28181 / ONVIF protocol to the user front-end (i.e. Figure 7 (55) Then the user front-end and the audio / video device front-end will stop uploading the user audio / video stream and the terminal audio / video stream respectively, and at the same time turn off the playback of the audio / video merged stream, thus ending the call.

[0084] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for recognizing a two-dimensional code, binding and interacting with a video, characterized in that, The device for two-dimensional code recognition binding and video interaction comprises the following steps: Step 1: A user front end scans a device two-dimensional code through a web browser supporting webRTC, the device two-dimensional code contains audio and video device terminal coding based on GB28181 or ONVIF protocol, and the device two-dimensional code and user coding are sent to a user management system, the user coding being pre-set in the user front end; Step 2: The user management system parses the device two-dimensional code to obtain an access link and audio and video device terminal coding, and the access link is returned to the user front end; The access link is used for registering or logging in a user account, and the audio and video device terminal coding is used for querying a specified audio and video device terminal; Step 3: The user front end parses the access link to obtain a registration / login page opened through the web browser supporting webRTC, and the user front end obtains a registration / login request through the registration / login page and sends the request to the user management system; the registration / login request comprises a user account, which is used for saving personal information, habits and viewing records of the user; Step 4: The user management system verifies the registration / login request, and if the user management system verifies that the registration / login request is passed, a binding request is returned to the user front end; In the process of verifying the user account, the user management system pre-stores accounts, judges whether the registration / login account is a pre-stored account, if yes, account information corresponding to the pre-stored account is called and sent to the user front end, and current device state information of the audio and video device terminal is sent to the user front end; If no, a new account is created for the registration / login account and stored in the user management system, and the new account information is sent to the user front end, and the current device state information of the audio and video device terminal is also sent to the user front end; Step 5: The user front end confirms whether to bind the audio and video device terminal, if yes, a determination binding request is generated and sent to the user management system; the binding request is used for letting the user confirm whether to perform associated binding on the user front end, and the binding request is visually displayed on the browser of the user front end; The user selects whether to perform associated binding through clicking on the browser of the user front end, if the user selects to perform associated binding, the binding request is sent to the user management system; If the user refuses to perform associated binding, the process of the web browser supporting webRTC for scanning and recognizing binding is ended; Step 6: When the user management system receives the determination binding request, a query request is generated and sent to a multi-protocol webRTC audio and video communication system together with the audio and video device terminal coding and the user coding; The query request is used for querying whether the audio and video device terminal associated with the audio and video device terminal coding is in an online state; If the audio and video device terminal associated with the audio and video device terminal coding is in the online state, the multi-protocol webRTC audio and video communication system performs associated binding between the audio and video device terminal corresponding to the audio and video device terminal coding and the user front end corresponding to the user coding; Step 7: The multi-protocol webRTC audio and video communication system performs associated binding between the audio and video device terminal corresponding to the audio and video device terminal coding and the user front end corresponding to the user coding; The multi-protocol webRTC audio and video communication system receives the query request, judges whether the audio and video device terminal is in an online state, if yes, associates and binds the audio and video terminal corresponding to the audio and video terminal coding and the user front end corresponding to the user coding, and if not, initializes the system; The audio and video device terminal periodically sends a heartbeat cycle to the multi-protocol webRTC audio and video communication system, and the multi-protocol webRTC audio and video communication system judges whether the audio and video device terminal is in an online state according to the heartbeat cycle; the audio and video device terminal automatically connects by sending a connection request to the signaling and session system through the GB28181 / onvif protocol, the signaling and session system records the audio and video device terminal coding and the online state of the audio and video device terminal under the GB28181 / onvif protocol, and the audio and video device terminal periodically reports its online state to the multi-protocol webRTC audio and video communication system through the heartbeat cycle; Step 8: The user front end sends a page call request to the signaling and session system through a webRTC-supported web browser; The user sends a page call request to the multi-protocol SIP signaling module of the signaling and session system through the browser of the user front end; the page call request is used to initiate audio and video interaction to the multi-protocol SIP signaling module; Step 9: The signaling and session system receives the page call request and obtains an audio and video session resource request; The multi-protocol SIP signaling module sends the audio and video session resource request to the stream pulling module, and the audio and video session resource request is used to initiate scheduling of the audio and video stream to the stream pulling module, so as to start the audio and video interaction between the user front end and the audio and video device terminal; Step 10: The signaling and session system obtains a scheduling stream pulling service resource instruction and a stream combining service resource instruction according to the audio and video session resource request; Step 11: The signaling and session system starts the stream pulling module and the stream combining module according to the scheduling stream pulling service resource instruction and the stream combining service resource instruction, the stream pulling module sends a front end stream pulling request and a terminal stream pulling request to the user front end and the audio and video device terminal respectively, the user front end sends a user audio and video stream based on the webRTC protocol to the stream pulling module after receiving the front end stream pulling request, the audio and video device terminal sends a terminal audio and video stream based on the GB28181 / ONVIF protocol to the stream pulling module after receiving the terminal stream pulling request, the stream combining module acquires the user audio and video stream and the terminal audio and video stream and synthesizes an audio and video stream according to a fixed picture layout, and the stream media distribution module acquires the audio and video stream and streams the audio and video stream to the audio and video device terminal based on the GB28181 / ONVIF protocol and to the user front end based on the webRTC protocol; The two-dimensional code recognition binding and video interaction device comprises a user front end, a multi-protocol webRTC audio and video communication system and an audio and video equipment terminal connected in communication.

2. The method of claim 1, wherein, The user front end is used for scanning and recognizing the device two-dimensional code through a webRTC-supported web page browser, obtaining an access link and an audio and video equipment terminal code by analyzing the device two-dimensional code, obtaining a registration / login page opened by the webRTC-supported web page browser by analyzing the access link, obtaining a registration / login request through the registration / login page, verifying the registration / login request, obtaining a binding request if the registration / login request is verified, confirming whether to bind the audio and video equipment terminal according to the binding request, generating a certain binding request if yes, generating a query request, the audio and video equipment terminal code and a user code according to the certain binding request, associating and binding the audio and video equipment terminal corresponding to the audio and video equipment terminal code and the user front end corresponding to the user code according to the query request, the audio and video equipment terminal code and the user code, initiating a page call request through the webRTC-supported web page browser, obtaining an audio and video session resource request according to the page call request, obtaining a scheduling pull stream service resource instruction and a distribution stream service resource instruction after the audio and video session resource request, starting the pull stream module and the stream module according to the scheduling pull stream service resource instruction and the distribution stream service resource instruction, calling the user audio and video stream based on the webRTC protocol and the terminal audio and video stream based on the GB28181 / ONVIF protocol according to the pull stream module, combining the user audio and video stream and the terminal audio and video stream into an audio and video stream according to the stream module, and playing the audio and video stream based on the webRTC protocol.

3. The method of claim 1, wherein, The multi-protocol webRTC audio and video communication system is used for associating and binding the audio and video equipment terminal corresponding to the audio and video equipment terminal code and the user front end corresponding to the user code.

4. The method of claim 1, wherein, The user management system is used for obtaining an access link and an audio and video equipment terminal code by analyzing the device two-dimensional code, opening a registration / login page according to the access link, obtaining a registration / login request through the registration / login page, verifying the registration / login request, obtaining a binding request if the registration / login request is verified, confirming whether to bind the audio and video equipment terminal according to the binding request, generating a certain binding request if yes, generating a query request according to the certain binding request, and sending the query request, the audio and video equipment terminal code and a user code to the multi-protocol webRTC audio and video communication system.

5. The method of claim 1, wherein, The pull stream module is configured to send a front-end pull stream request and a terminal pull stream request to a user front end and an audio / video device terminal respectively, the stream merging module is configured to acquire user audio / video stream and terminal audio / video stream and synthesize audio / video stream merging according to a fixed picture layout, and the stream media distribution module is configured to acquire the audio / video stream merging and play out the audio / video stream merging to the audio / video device terminal based on the GB28181 / ONVIF protocol and to the user front end based on the webRTC protocol.

6. The method of claim 1, wherein, The audio / video device terminal is configured to acquire terminal audio / video stream based on the GB28181 / ONVIF protocol, synthesize audio / video stream merging according to a fixed picture layout based on the user audio / video stream and the terminal audio / video stream, and play the audio / video stream merging based on the GB28181 / ONVIF protocol.

Images