Audio-video peripheral collaborative operation system and operation method

Abstract

The invention discloses an audio-video peripheral collaborative operation system and an operation method, and the system proposes an audio-video host which comprises an audio-video processing unit, a micro-control unit for operating audio-video peripheral collaborative operation, at least one connection interface and a data processing unit. The audio-video host is connected with the peripheral device through the connection interface according to a communication protocol, detects a photographic lens and a microphone of the peripheral device, receives videos and audios from the peripheral device according to the communication protocol after obtaining the authority of accessing the photographic lens and the microphone, processes the videos and audios through the data processing unit, and sends the processed videos and audios to the peripheral device. The micro-control unit generates the audio-video data provided for the audio-video processing unit, then the display can display the video, and the loudspeaker can play the audio, so that the video conference can be executed by the audio-video host without a photographic lens and a microphone.

Description

Technical Field

[0001] The specification discloses a system using peripheral photography and audio receiving devices, in particular, an audio-visual peripheral cooperative operating system and operation method for an audio-visual host to use the audio-visual modules on its peripheral devices to execute video purposes. Prior art

[0002] Common audio and video hosts (such as TVs, set-top boxes, etc.) do not have cameras or microphones installed due to cost or other considerations. For example, these peripherals are not needed for general applications, or are only used to initialize the equipment during initial setup and startup, so the usage rate is not high; or users will purchase equipment that meets their needs. Therefore, except for a few high-end models, most audio and video hosts do not have preset cameras or microphones.

[0003] Because the above peripherals are not set up, the traditional audio and video host cannot expand more advanced functions. For example, an audio and video host with a microphone can use the microphone to receive the sound from the surround sound, and then perform the correction of the speaker position according to the sound information, or the audio and video host can use a camera to shoot images, and then use the image processing to determine the placement of items in the home.

[0004] Therefore, when users use the video host to execute specific applications, additional devices are needed to assist. For example, when conducting a video conference, the video host needs to be connected to a camera and a microphone, but due to the problem of cross-operating systems, it is difficult to set up.

[0005] In the prior art, the audio and video host is provided with a connection interface for expanding peripherals, such as a Universal Serial Bus (USB), which can be used to expand the peripheral devices of the audio and video host. However, because the compatibility problem of the driver is still very large, even if the Universal Serial Bus is adopted, the expandability of the audio and video host still has considerable problems. Summary of the invention

[0006] In response to the problem that traditional audio and video hosts that provide audio and video signal processing lack video and audio peripherals, the manual proposes an audio and video peripheral collaborative operating system and operation method, providing a cross-platform integrated audio and video peripheral solution so that the audio and video host can effectively use the video and audio functions of the peripheral devices to achieve purposes such as audio and video communication and video conferencing. Its technical concept is to use the communication protocol of video conferencing to communicate between audio and video hosts and peripheral devices on different operating platforms.

[0007] According to an embodiment of the audio-visual peripheral collaborative operating system, an audio-visual host is mainly provided, which is provided with an audio-visual processing unit for processing audio-visual data, and includes a micro control unit for running audio-visual peripheral collaborative operations, at least one connection interface, and a data processing unit for processing data transmitted through at least one connection interface.

[0008] Furthermore, the operation method includes that the audio and video host connects to at least one peripheral device through at least one connection interface using a communication protocol, and detects one or more cameras and one or more microphones of each peripheral device. After that, after obtaining the permission to access the one or more cameras and one or more microphones, the audio and video host can receive video and audio from the peripheral device through at least one connection interface using the communication protocol, and after being processed by the data processing unit, the micro control unit generates audio and video data provided to the audio and video processing unit, and after being processed by the audio and video processing unit, the audio and video data can be connected to the display of the audio and video host to display the video and the speaker to play the audio.

[0009] Preferably, when the audio / video host and the peripheral devices are in the same local area network or area, they are connected to each other via a wireless local area network or Bluetooth communication protocol.

[0010] Preferably, when the audio and video host and the peripheral devices are not in the same local area network, a software program using an interactive connection establishment communication protocol or a web instant communication protocol is used to establish a connection.

[0011] Preferably, the connection interface may be a wireless or wired communication interface for receiving videos captured by multiple cameras of one or more peripheral devices and audio received by multiple microphones.

[0012] Furthermore, multiple microphones can form a microphone array to obtain audio from microphones at different positions and perform multi-channel beamforming to track the sound source. The microcontroller in the audio and video host obtains the volume received by each microphone and can locate multiple microphones according to the audio intensity received by each microphone. Furthermore, after the positioning of multiple microphones is completed, the position, direction and gain of multiple microphones can be adjusted according to the user's position.

[0013] Furthermore, the microcontroller in the video host obtains the images captured by each camera lens, and can compare the video images captured by each camera lens through image processing technology to determine the position of each camera lens. In this way, in the video host, stereo detection is performed on multiple images captured by multiple cameras to determine the position of the user.

[0014] Furthermore, the identification data can be used to log in to the server program of the audio and video host and the audio and video program of the peripheral device respectively, thereby establishing a connection between the audio and video host and the peripheral device, and allowing the audio and video host to obtain access to one or more cameras and one or more microphones of the peripheral device.

[0015] Furthermore, the audio and video host also executes a calibration program for calibrating the video and audio, which can determine the position of one or more microphones and one or more cameras on the peripheral device, the position of the speakers connected to the audio and video host, and adjust the post-processing and volume of the speakers, or adjust the brightness and color of the picture displayed on the monitor.

[0016] In one application, the audio and video host and the peripheral devices form a first conference terminal, which can establish a conference connection with a second conference terminal at the other end.

[0017] In order to further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the system architecture of the audio-visual peripheral collaborative operating system is shown;

[0019] Figure 2 An embodiment diagram of an audio-visual peripheral collaborative operating system is shown;

[0020] Figure 3 A flowchart of an embodiment of the operation of the audio-visual peripheral cooperative operating system is shown;

[0021] Figure 4 A schematic diagram showing an embodiment of a handshake connection between an audio and video host and a peripheral device;

[0022] Figure 5 A flow chart of an embodiment of establishing a conference connection is shown;

[0023] Figure 6 A flow chart showing an embodiment of establishing a connection through a network address translation host;

[0024] Figure 7 A flowchart of an embodiment of a multi-channel application in an audio-visual peripheral collaborative operating system is shown; and

[0025] Figure 8 A flow chart of an embodiment of a multi-lens application in an audio-visual peripheral collaborative operating system is shown. DETAILED DESCRIPTION

[0026] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0027] It should be understood that, although the terms "first", "second", "third", etc. may be used herein to describe various components or signals, these components or signals should not be limited by these terms. These terms are mainly used to distinguish one component from another component, or one signal from another signal. In addition, the term "or" used herein may include any one or more combinations of the associated listed items depending on the actual situation.

[0028] The specification proposes an audio-visual peripheral collaborative operating system and operation method, and proposes a solution for using the audio-visual functions of peripheral devices for audio-visual hosts (such as set-top boxes, televisions, etc.) that are not equipped with audio-visual peripherals. One of the technical solutions is to use the communication protocol of video conferencing to communicate between audio-visual hosts and peripheral devices on different operating platforms, so that the audio-visual host can integrate audio-visual peripherals across platforms, allowing the audio-visual host to effectively use the video and audio functions of the peripheral devices to achieve purposes such as audio-visual communication and video conferencing.

[0029] According to the implementation method of the audio-visual peripheral collaborative operating system, you can refer to Figure 1 The system architecture diagram shown shows an audio and video host 10 for processing audio and video signals, such as a set top box, an audio and video processing module in a smart TV, a computer system or a specific audio and video device. The audio and video host 10 is provided with an audio and video processing unit 103 for processing audio and video data, and related input and output circuits, including an input interface 101 for receiving audio and video data 11 and an output interface 105 for outputting audio and video content to a display 12 and a speaker 13.

[0030] Furthermore, the audio and video host 10 is also provided with an operation module for processing data generated by the collaborative operation of the audio and video peripherals, which has relevant processing circuits and software programs, such as a micro control unit 106 for processing various external information and instructions as shown in the figure. The micro control unit 106 is electrically connected to the audio and video processing unit 103, so that the video and audio processed by the micro control unit 106 can be processed by the audio and video processing unit 103 (including audio and video encoding and decoding) and then output to the display 12 and the speaker 13 through the output interface 105.

[0031] The microcontroller unit 106 is used to control the operation of the audio and video host 10, and can provide data received through a specific interface to the audio and video processing unit 103 for processing. According to the embodiment of the audio and video host 10 shown in the figure, the audio and video host 10 provides a connection interface 109 (the number can be one or more) for connecting to an external device and a corresponding control circuit, i.e., a data processing unit 108. The connection interface 109 can refer to various wired and wireless communication interfaces or connection terminals, such as a universal serial bus (USB), Bluetooth (Bluetooth), etc. TM )Communication interface and wireless network (WiFi TM ) interface, or other interfaces under industrial standards (such as various microphone connectors), and the actual application is not limited to the interfaces listed above. The micro control unit 106 is electrically connected to the data processing unit 108, and the data processing unit 108 is used to process the data received through the connection interface 109, so that the video and audio data received through the connection interface 109 can be processed by the data processing unit 108 and provided to the audio and video processing unit 103 for processing and output.

[0032] According to the illustrated embodiment, the audio and video host 10 and the peripheral device 15 are different operating platforms running different operating systems. The audio and video host 10 is connected to one or more peripheral devices 15 via a connection interface 109, and uses a data processing unit 108 to process data transmitted via the connection interface 109. The peripheral device 15 can be a powerful computer device with audio and video processing capabilities, such as a smart phone, a tablet computer or a personal computer, or it can be a peripheral device with a single function, such as a camera 17 and a microphone 18 equipped with a photographic module and an audio receiving module, respectively, as shown in the figure.

[0033] For example, as shown in the figure, the peripheral device 15 connected to the audio and video host 10 via Bluetooth or wireless network communication technology is an electronic device having a camera module 151, a sound receiving module 153 and a communication module 155. The peripheral device 15 can be a variety of computer devices such as a smart phone, a tablet computer and a notebook computer. The figure ignores the necessary circuit components, such as a processor, a memory and related peripherals. The camera module 151 of the peripheral device 15 mainly includes a lens, a photosensitive element and an image processing circuit, the sound receiving module 153 mainly includes a microphone and an audio processing circuit, and the communication module 155 can be a Bluetooth TM ) module, wireless network (WiFi TM ) module or other wireless communication module.

[0034] The peripheral device 15 shown is a handheld device such as a commonly used smart phone or tablet computer. Such smart devices are equipped with multiple microphones to perform the functions of sound reception and noise reduction. They may also be equipped with multiple cameras to provide photography functions with multiple focal lengths such as macro, telephoto, standard and wide angle. Therefore, when the audio-visual peripheral cooperative operating system is running, the audio-visual host 10 can be connected to the peripheral device 15 through the connection interface 109 with a specific communication protocol, and with the software technology for processing the communication protocol, the video and audio received by the peripheral device 15 in a specific format are obtained. After the data is processed by the data processing unit 108, it is provided by the micro control unit 106 to the audio-visual processing unit 103 for audio-visual codec processing, and then output to the display 12 through the output interface 105 to display the screen, and the speaker 13 plays the sound.

[0035] You can continue to refer to Figure 2 The diagram shows an implementation example of an audio-visual peripheral collaborative operating system.

[0036] One of the operating methods of the audio-visual peripheral collaborative operating system is to run a server program (server program) 20 for processing video and / or audio transmitted by peripheral devices on the audio-visual host 10, which is executed by the micro control unit 106, in which a program for processing audio-visual data can be installed and executed, such as the audio-visual program (audiovisual program) 201 shown in the figure, which can be an audio-visual program pre-installed in the audio-visual host 10, or a specific audio-visual program installed through online hardware real-time update (over-the-air, OTA) method.

[0037] According to one embodiment, the audio and video program 201 can be a common video conferencing program that has been formed into a packaged software, or a software program (such as a web browser) that uses a web browser to execute a web real-time communication protocol (Web Real-Time Communication, WebRTC). In particular, this type of audio and video program 201 can be a lightweight program that only needs to process one-way audio and video data (the audio and video host 10 only needs to receive the audio and video data generated by the peripheral device), and can also be installed to execute a calibration program 203 for calibrating video and audio. The calibration program 203 can be used to determine the position of one or more microphones (213, 223) and one or more camera lenses (212, 222) on the peripheral device 1 21 and the peripheral device 2 22, the position of the speaker 27 connected to the audio and video host 10, adjust the post-processing (such as amplification power, high and low frequency adjustment, etc.) and volume of the speaker, and adjust the brightness and color of the picture displayed on the display 25.

[0038] Furthermore, the server program 20 executed in the audio and video host 10 can process the audio and video data transmitted by different peripheral devices (21, 22), and then the decoded output image is displayed on the display 25, and the sound is played through the speaker 27. Therefore, when a single user conducts a video conference, the processing of the audio and video host 10 can provide video conferencing or audio and video communication of multiple images at the same time.

[0039] According to the system embodiment, multiple image sources and audio sources can be provided when a single user conducts a video conference. Figure 2 In the embodiment shown, the video host 10 can be connected to a plurality of peripheral devices, including a peripheral device 1 21 and a peripheral device 2 22. The peripheral device 1 21 is provided with a camera lens 1 212 and a microphone 1 213, wherein the video program 1 211 is executed, and the video program 1 211 can be interconnected with the video program 201 executed in the video host 10 and transmit the video produced by the camera lens 1 212 and the audio produced by the microphone 1 213. The peripheral device 2 22 is provided with a camera lens 2 222 and a microphone 2 223, wherein the video program 2 221 is executed, and similarly, the video program 2 221 can be interconnected with the video program 201 executed in the video host 10 and transmit the video data produced by the camera lens 2 222 and the microphone 2 223.

[0040] In this way, the audio and video host 10 establishes connections with the audio and video program 1 211 of the peripheral device 1 21 and the audio and video program 2 221 of the peripheral device 2 22 through the server program 20, and obtains the permission to access the audio and video modules (individual cameras and microphones) in each device (peripheral device 1 21 and peripheral device 2 22), allowing users to realize video conferencing or audio and video communication with multiple audio and video sources through the audio and video peripheral collaborative operating system.

[0041] It is worth mentioning here that in the audio and video peripheral collaborative operating system, by using audio and video programs (such as audio and video program one 211 and audio and video program two 221) executed in multiple peripheral devices (such as peripheral device one 21 and peripheral device two 22), multiple cameras and microphones on multiple peripheral devices can be used, so that the audio and video peripheral collaborative operating system can realize the application of audio and video communications with multiple image sources and multiple sound channels, among which a microphone array can be realized.

[0042] In another embodiment, a single peripheral device may have multiple cameras and multiple microphones, which can also allow users to use multiple images and multiple channels for video conferencing or audio and video communication. Taking a smartphone as an example, it is usually equipped with 2 to 4 microphones, including at least one microphone on the top of the phone for noise reduction and at least one microphone on the bottom of the phone for calls, or a microphone on the main lens can be added to realize a microphone array.

[0043] In this way, the audio and video program executed in multiple peripheral devices or a single peripheral device can obtain the audio of microphones in different positions, obtain a multi-channel sound source, and perform multi-channel beamforming to track the sound source. The beamforming technology determines the direction of the sound source (the speaking person's mouth) based on the time difference between the sound and audio signals arriving at different microphones, which enables the audio and video program 201 executed in the audio and video host 10 to determine the sound source and ignore other sound sources.

[0044] According to the description of the above embodiment of the audio-visual peripheral collaborative operating system, the audio-visual host 10 can be connected to one or more peripheral devices (21, 22), and can flexibly use one or more cameras and one or more microphones owned by each peripheral device. Each peripheral device (21, 22) is connected to the audio-visual host 10 in a point-to-point manner. In addition to transmitting video and audio from the peripheral devices (21, 22) to the audio-visual host 10 during audio-visual conferencing or audio-visual communication, in one embodiment, the audio-visual host 10 can also reversely transmit data, text, sound and images to the peripheral devices (21, 22), and there is no need to limit the connection software installed in the peripheral devices, as long as both parties follow the same communication protocol.

[0045] The method of running the audio-visual peripheral collaborative operating system can be referred to Figure 3 In the flowchart of the embodiment shown, when the method is running, the audio and video host and the peripheral devices can be in the same local area network (LAN) or in different network domains, and the audio and video host and the peripheral devices run the same communication protocol, that is, both parties execute the agreed video conferencing program.

[0046] When the audio and video host and the peripheral devices are in the same local area network or in a nearby area, the connection method can be a wireless local area network connection or a Bluetooth pairing connection method. After the audio and video host and the peripheral devices start the corresponding audio and video programs, they can detect each other's existence and pair up. After the connection is completed, the peripheral devices execute the video conferencing program and establish a video connection with the audio and video host through the agreed name or number.

[0047] When the video host and peripheral devices are not in the same local area network, that is, they are connected via a wide area network (WAN), the video host and peripheral devices can use the Interactive Connectivity Establishment (ICE) communication protocol or the Web Real-time Communication Protocol (WebRTC) software program. When the video host is turned on, it can first obtain its own public network address, and the peripheral devices also have their own public network addresses after turning on, and then establish a connection through the agreed name or number.

[0048] After the video host and the peripheral device start the corresponding video program, they can detect each other's existence and establish a connection between the video host and the peripheral device through the above connection method (step S301). According to one embodiment, a barcode recording the host name and network address, such as a QR code, can be printed on the body of the video host, so that the peripheral device can automatically connect by scanning the barcode after opening the camera lens.

[0049] One way to establish a connection can refer to Figure 4 The diagram shows an embodiment of a handshake connection between the audio and video host and the peripheral devices.

[0050] The video host 41 and the peripheral device 42 shown therein are in different network domains. The devices in different network domains may already have private network addresses (private IP addresses) configured by the network domains. However, to establish a connection, they can obtain different public network addresses (public IP addresses) from the session traversal utilities for NAT server (STUN) 40 for network address translation (NAT) shown in the figure, or establish a connection through a relay traversal server (traversal using relay NAT, TURN) 45 using relay network address translation, and the connection is completed. Figure 3 Step S301, the operation details of which can be found in Figure 5 .

[0051] The process of establishing a connection between the audio and video host 41 and the peripheral device 42 via the conference penetration application server 40 can be referred to Figure 5 A flow chart of an embodiment of establishing a conference connection is shown.

[0052] At the beginning, the audio and video host 41 and the peripheral device 42 complete the preparation before the audio and video collaborative operation, including installing and setting the corresponding communication protocol or audio and video program (step S501). Then, the audio and video host 41 sends a request packet to the conference penetration application server 40 to obtain a public network address (step S503), and the peripheral device 42 also sends a request packet to the conference penetration application server 40 to obtain another public network address (step S505). If the audio and video host 41 or the peripheral device 42 fails to connect to the conference penetration application server 40 on the network, it can then request the relay penetration server 45 to obtain a public network address.

[0053] Afterwards, when both the audio and video host 41 and the peripheral device 42 obtain the public network address, the peripheral device 42 requests the audio and video host 41 to establish a conference connection (step S507), wherein the public network addresses of each other can be exchanged through a handshake process, and then a connection is established through the audio and video programs executed by each of them (step S509). In the process of establishing a connection between the audio and video host 41 and the peripheral device 42, the audio and video host 41 can operate as a conference host to start a meeting, and then the peripheral device 42 can request the audio and video host 41 to establish a conference connection. Taking common conference software as an example, the audio and video host 41 operates as a conference host and can receive requests from one or more peripheral devices 42 to establish a conference connection, that is, to start an online meeting, and provide a conference link (or conference room code), which can be represented by a barcode. After the peripheral device 42 scans the barcode or enters the conference room code, a conference connection can be established between the audio and video host 41 and one or more peripheral devices 42, and the meeting is completed. Figure 3 Step S301.

[0054] In the preparation before the audio / video host and the peripheral device establish a connection in step S501, Figure 1 In the illustrated embodiment, a server program is executed in the audio and video host to manage user accounts. In specific applications, a user (having user identification data) needs to log in to the audio and video host. The peripheral device can be a smart device that has logged in to the user account, so that the audio and video programs respectively executed in the audio and video host and the peripheral device can establish a connection based on the same user identification code. The audio and video host can obtain permission to use the camera and microphone in the peripheral device with the user's authorization (for example, authorization can be obtained through specific identification data).

[0055] For example, a peripheral device such as a user's mobile device, the user logs into a video program in the mobile device with a user identification data (user ID) (such as Figure 2 The same user identification data is used to log in to the server program in the audio and video host (such as Figure 2 After that, the video program in the mobile device establishes a connection with the server program in the video host. Because both of them log in the same user identification data, the video host can obtain the permission to use the video module (one or more cameras and / or one or more microphones) in the mobile device, so that the video host can obtain the video data generated by the mobile device. Conversely, in one embodiment, the video host can also reversely transmit signals, data, text, sound and / or images through the connection established with the mobile device.

[0056] Then, in Figure 3In step S303, the server program in the audio and video host can receive information from the peripheral device through connection, and detect the audio and video modules that the peripheral device can support, including one or more cameras and one or more microphones. Through user authorization, the audio and video host can obtain the permission to access the audio and video modules of the peripheral device and initialize the audio and video modules of the system and the peripheral device (step S303).

[0057] At this time, if Figure 3 As shown in step S305, a session link is established, and audio and video communication or conference may begin. The audio and video host begins to receive the video and audio generated by the peripheral devices (step S307), and can also perform audio and video calibration through the calibration program (step S309). The received video and audio are processed by the data processing unit, and then the microcontroller generates audio and video data provided to the audio and video processing unit in the audio and video host. After further processing by the audio and video processing unit, the optimized and calibrated audio and video content is output through the output interface (step S311). At this time, the meeting starts, and a video conference is conducted according to the above calibration results, in which the video is displayed on the display connected to the audio and video host, and the audio is played on the speaker connected to the audio and video host (step S313). The meeting can be ended after completion.

[0058] Taking a multi-person video conference as an example, the above embodiment shows that the end user uses an audio and video host and peripheral devices to implement a first conference terminal, and establishes a conference connection with a second conference terminal through a specific conference host. The end user can use a general conference system, or a conference terminal implemented by an audio and video peripheral collaborative operating system proposed in the specification. In this way, the audio and video host of the first conference terminal displays the video received from at least one peripheral device on the connected display, and plays the audio received from at least one peripheral device on the connected speaker, so as to perform a video conference with the second conference terminal or more conference terminals.

[0059] Examples can be referenced Figure 6 The process of establishing a connection through a network address translation host is shown. The conference host 60 shown in the figure can be used to provide multi-party conference connection services and to manage the authentication of users at each end and the establishment of conference links. The users at each end may be in a specific local area network and need to convert the private network address to a public network address through the network address translation (NAT) mechanism. One of the communication protocols for establishing the conference connection can be the Web Real Time Communication Protocol (WebRTC).

[0060] For example, terminal user A 61 and / or terminal user B 62 can be conference terminals implemented using the audio-visual peripheral collaborative operating system. Terminal user A 61 is in a network domain and has a private network address. Its network address can be converted into a public network address through a network address conversion host A 63; similarly, terminal user B 62 can also be converted into a public network address through a network address conversion host B 64 in the network where it is located.

[0061] When establishing a conference connection, terminal user A 61 requests the conference host 60 to establish a connection with terminal user B 62 (step S601). The connection request will submit terminal user A's information, including the Session Description Protocol (SDP), to the conference host 60, and then the connection request and the SDP of terminal user A 61 will be transmitted to terminal user B 62 through the conference host 60 (step S603). When terminal user B 62 receives the connection request and the SDP of terminal user A 61, it will respond to the SDP (step S605), in which it will respond to the information of terminal user B 62, including the SDP of terminal user B 62, and then the SDP of terminal user B 62 will be forwarded to terminal user A 61 through the conference host 60 (step S607). In this way, terminal user A 61 and terminal user B 62 can obtain each other's connection information and conduct a point-to-point conference connection. The two parties may use the Interactive Connection Establishment (ICE) communication protocol or the Web Real-Time Communication (WebRTC) protocol for subsequent communication and transmit audio and video content under the agreed communication protocol.

[0062] It is worth mentioning that, through the above connection process, a point-to-point connection can be established between the end user A 61 and the end user B 62, but if the connection fails, it can be switched to use a relay penetration server (TURN) for connection.

[0063] The audio-visual peripheral cooperative operating system enables an audio-visual host without an audio-visual peripheral to use the audio-visual modules on the peripheral device, including a camera module and a radio module. When the peripheral device used by the audio-visual host has multiple radio modules, or the audio-visual host is used with multiple peripheral devices at the same time and thus obtains access rights to multiple radio modules at the same time, the operation can be performed as follows: Figure 7 The flowchart of an embodiment of a multi-channel application in an audio-visual peripheral collaborative operating system is shown.

[0064] After the connection between the audio and video host and one or more peripheral devices is completed, and the audio and video host obtains the permission to access the audio and video modules in one or more peripheral devices, the video conference can be started (step S701). At this time, the adjustment program can be started to execute the audio and video program in the audio and video host to use the speakers connected to the audio and video host to produce sound in turn with multiple audio channels (step S703). For example, the audio and video host is connected to a surround sound system with 5.1 channels, and each channel can be controlled to produce audio in turn.

[0065] At this time, the audio and video host controls the operation of the sound receiving modules of one or more connected peripheral devices, and uses multiple microphones (each microphone is set with a microphone identification code) to simultaneously receive the sound generated by the surround sound system (step S705). The micro control unit in the audio and video host obtains the volume received by each microphone (step S707), and can locate the microphone according to the audio intensity received by each microphone (step S709).

[0066] Then, beamforming is performed based on the audio received from the multi-channel (step S711). Through the calculation of the audio and video host microcontroller unit, the location of the sound source can be determined based on the time difference between the audio signal reaching different microphones to optimize the relative volume (step S713). For example, the audio and video host can locate the position of each speaker based on the surround audio signal received by the sound receiving module, and repeat the above procedure to correct the speaker position and the microphone position to obtain a better sound effect. One of the applications is that when a user is conducting a video conference, the position, direction and gain of multiple microphones can be adjusted according to the user's position (the user's position can be determined through image processing).

[0067] Using the audio-visual peripheral collaborative operating system, it can be achieved Figure 8 The flowchart of the multi-lens application embodiment is shown.

[0068] When a video conference is started with a video host connected to multiple cameras (step S801), the video conference can rotate the images taken by multiple cameras. In one embodiment, the video host micro control unit obtains the video generated by each camera module (step S803), and compares each video image through image processing technology to determine the lens position (step S805). For example, after the algorithm running in the micro control unit obtains the image information, it can normalize the picture and enlarge or reduce it to the same size image, so as to compare the difference and similarity of the videos taken by each camera lens, which is used to determine the position of each lens, such as determining the front lens and rear lens of the mobile phone, whether it is a wide lens or a telephoto lens, etc.

[0069] In this way, the microcontroller unit in the audio and video host obtains the characteristic value of each video image after calculation (step S807), and groups the video images according to the characteristics of each lens (step S809). For example, after obtaining the shooting direction of each lens on the smartphone, the camera lens can be divided into the front lens and the rear lens according to the shooting direction. Afterwards, the audio and video host can use the multiple images obtained at the same time to perform stereo detection, analyze the layout in the image, and determine the position of the user performing the video conference (step S811). Therefore, before or during the video conference, the microphone can be adjusted according to the user's position, including the microphone position, direction and gain, etc. (step S813).

[0070] According to the embodiment, when the user uses only a single lens for video conferencing, when the audio and video host obtains the image information, the layout in the picture can be analyzed to obtain the ambient brightness, and the picture brightness of each camera lens can be adjusted. On the other hand, when one end of the video conference uses more than two lenses, the audio and video host can use the images taken by multiple lenses for stereo detection, and further determine the relative distance of all objects in the environment. The error of the microphone performing beamforming can be corrected accordingly. For example, when the position of the speaker in the picture is accurately identified by image recognition technology (artificial intelligence module can be used), a more accurate sound field can be adjusted. On the other hand, when the camera cannot determine the depth of field, the audio and video host can drive each speaker to emit audio, and then receive the sound through the microphone. The time difference between the microphones is calculated based on the audio signal, and the position of the microphone can be inferred accordingly. Therefore, the audio and video peripheral collaborative operating system proposed in the specification can perform environmental detection based on the audio received by the multiple microphones connected to the audio and video host and the images taken by multiple cameras, and can achieve the purpose of mutual correction between the microphone and the camera lens.

[0071] In summary, according to the embodiments of the above-mentioned audio-visual peripheral collaborative operating system and operating method, the system enables an audio-visual host that is not originally equipped with a camera and a microphone, or an audio-visual host that needs to be expanded with more microphones and cameras, to directly use the audio-visual module of the peripheral device to perform video conferencing. The cross-platform communication technology is used to connect the audio-visual host and the peripheral devices, so that the audio-visual host does not need to increase the additional cost of the peripherals, and can provide more optimized audio-visual video services. Moreover, compared with the traditional device with a fixed camera lens and a microphone, the proposed audio-visual peripheral collaborative operating system and operating method provide a more flexible and expandable solution.

[0072] The contents disclosed above are only preferred feasible embodiments of the present invention and are not intended to limit the protection scope of the application. Therefore, all equivalent technical changes made by applying the contents of the specification and drawings of this application are included in the protection scope of this application.

[0073] Description of Reference Numerals

[0074] 10: Audio and video host

[0075] 11: Audio and video data

[0076] 101: Input interface

[0077] 103: Audio and video processing unit

[0078] 12: Display

[0079] 13: Speaker

[0080] 105: Output interface

[0081] 106: Micro control unit

[0082] 108: Data processing unit

[0083] 109: Connection interface

[0084] 17: Camera

[0085] 18: Microphone

[0086] 15: Peripheral devices

[0087] 151: Photography Module

[0088] 153: Radio module

[0089] 155: Communication module

[0090] 20: Server Program

[0091] 201: Audio and Video Program

[0092] 203: Adjustment Procedure

[0093] 21: Peripheral device 1

[0094] 212: Camera Lens 1

[0095] 213: Microphone 1

[0096] 211: Audio and Video Program 1

[0097] 22: Peripheral Device 2

[0098] 222:Photographic lens 2

[0099] 223: Microphone 2

[0100] 221: Audio and Video Program 2

[0101] 25: Display

[0102] 27: Speaker

[0103] 40: Conference penetration application server

[0104] 45: Relay penetration server

[0105] 41: Audio and video host

[0106] 42: Peripheral devices

[0107] 60: Conference Host

[0108] 61: End User A

[0109] 63: Network Address Translation Host A

[0110] 62: End User B

[0111] 64: Network address translation host B

[0112] Steps S301 to S313: Operational flow of the audio-visual peripheral collaborative operating system

[0113] Steps S501 to S509 Conference connection process

[0114] Steps S601 to S607: Process of establishing a connection through a network address translation host

[0115] Steps S701 to S713 Multi-channel application process

[0116] Steps S801 to S813 Multi-lens application process

Claims

1. A video and audio peripheral collaborative operating system, comprising: A video and audio host, which includes a video and audio processing unit, a micro control unit for running video and audio peripheral collaborative operations, at least one connection interface, and a data processing unit for processing data transmitted through the at least one connection interface; Wherein the video and audio host is connected to at least one peripheral device through the at least one connection interface by a communication protocol, and detects one or more camera lenses and one or more microphones of the at least one peripheral device; After obtaining the permission to access the one or more camera lenses and the one or more microphones, the video and audio host receives video and audio from the at least one peripheral device through the at least one connection interface by the communication protocol. After being processed by the data processing unit, video and audio data provided to the video and audio processing unit is generated through the micro control unit. After being processed by the video and audio processing unit, the video is displayed on a display connected to the video and audio host, and the audio is played through a speaker.

2. The video and audio peripheral collaborative operating system according to claim 1, wherein, When the video and audio host and the peripheral device are in the same local area network or region, they are connected to each other by a wireless local area network or Bluetooth communication protocol.

3. The video and audio peripheral collaborative operating system according to claim 1, wherein, When the video and audio host and the peripheral device are not in the same local area network, a software program using an interactive connection to establish a communication protocol or a web instant messaging protocol is used to establish a connection.

4. The video and audio peripheral collaborative operating system according to claim 1, wherein the at least one connection interface is a wireless or wired communication interface for receiving video captured by a plurality of camera lenses of one or more peripheral devices and audio received by a plurality of microphones.

5. The video and audio peripheral collaborative operating system according to claim 4, wherein the micro control unit in the video and audio host obtains the volume received by each microphone, locates the plurality of microphones according to the audio intensity received by each microphone, and after completing the positioning of the plurality of microphones, adjusts the position, direction and gain of the plurality of microphones according to the user's position.

6. The video and audio peripheral collaborative operating system according to claim 4, wherein the micro control unit in the video and audio host obtains the images captured by each camera lens, and compares the video images captured by each camera lens through image processing technology to determine the positions of the respective camera lenses.

7. The video and audio peripheral collaborative operating system according to any one of claims 1 to 6, wherein, The server program of the video and audio host and the video and audio programs of the at least one peripheral device are logged in with identification data to establish a connection between the video and audio host and the at least one peripheral device, and enable the video and audio host to obtain the permission to access the one or more camera lenses and the one or more microphones of the at least one peripheral device.

8. The audio-visual peripheral collaborative operating system according to claim 7, wherein a video conferencing program or a software program implementing a web instant messaging protocol is executed in the audio-visual host; the audio-visual host also executes a calibration program for calibrating video and audio, and determines, through the calibration program, the positions of the one or more microphones and the one or more camera lenses on the at least one peripheral device, the position of the speaker connected to the audio-visual host, adjusts the post-processing and volume of the speaker, and adjusts the brightness and color of the image displayed on the display.

9. The audio-visual peripheral collaborative operating system according to any one of claims 1 to 6, wherein the audio-visual host and the at least one peripheral device form a first conference terminal and establish a conference connection with a second conference terminal; wherein the audio-visual host of the first conference terminal displays, on the display, the video received from the at least one peripheral device and plays, through the speaker, the audio received from the at least one peripheral device to conduct a video conference.

10. A method for operating an audio-visual peripheral collaborative operating system, which runs in an audio-visual host comprising: connecting, through at least one connection interface of the audio-visual host, at least one peripheral device using a communication protocol and detecting one or more camera lenses and one or more microphones of the at least one peripheral device; after obtaining access to the one or more camera lenses and the one or more microphones, the audio-visual host receives video and audio from the at least one peripheral device through the at least one connection interface using the communication protocol; and generating, by a micro control unit of the audio-visual host, audio-visual data provided to an audio-visual processing unit, and after being processed by the audio-visual processing unit, displaying the video on a display connected to the audio-visual host and playing the audio through a speaker.

Images