Audio and video equipment and corresponding multi-equipment data transmission method

Abstract

The embodiment of the invention relates to the technical field of communication, and discloses an audio and video device and a corresponding multi-device data transmission method, comprising the following steps: a master device acquires a video stream from at least one slave device so as to acquire a multi-channel video stream formed by the video stream of the master device and the video stream of the at least one slave device; splicing the multiple paths of video streams to obtain a spliced video stream; the master device obtains the audio stream from the at least one slave device so as to obtain at least one audio stream formed by the audio stream of the master device and / or the audio stream of the at least one slave device; and adding at least one audio stream to the spliced video stream to obtain a synthetic video stream to be displayed. Through establishing a joint cooperative transmission channel among a plurality of audio and video devices, the fast sharing of audio data and video data is realized, and the efficiency and quality of conference communication are greatly improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of audio and video transmission, and in particular to an audio and video device and a corresponding multi-device data transmission method. Background Art

[0002] With the widespread adoption of various electronic devices, audio and video devices with cameras and microphones, such as smartphones, tablets, televisions, and camcorders, have become widely used in various fields, including life, education, and work. In today's digital age, people increasingly use digital devices to record their lives and transmit information, leading to the rapid development of related technologies.

[0003] Currently, audio and video devices typically operate independently, meaning recorded audio and video data is stored separately on different devices, lacking integration and unification. This is especially true in situations like meetings and training sessions, where multiple devices are involved, making the processing and integration of audio and video data crucial. While some audio and video devices can live stream and record, they operate solely on a single device and are unable to integrate audio and video data from multiple devices into a single, complete stream.

[0004] Therefore, it is necessary to provide a transmission tool and technology that can unite these devices so that they can work together to achieve the expected effect of the meeting. Summary of the Invention

[0005] The embodiments of the present application provide an audio and video device and a corresponding multi-device data transmission method to solve the problem in the prior art that only a single device can be operated and the audio and video data of multiple devices cannot be integrated into a complete audio and video stream.

[0006] The purpose of the embodiment of the present invention is achieved through the following technical solutions:

[0007] To solve the above technical problems, in a first aspect, an embodiment of the present invention provides a multi-device data transmission method, comprising:

[0008] The master device acquires a video stream from at least one slave device to obtain a multi-channel video stream consisting of the video stream of the master device itself and the video stream of at least one slave device;

[0009] Splicing multiple video streams to obtain a spliced ​​video stream;

[0010] The master device obtains the audio stream from the at least one slave device to obtain at least one audio stream composed of the audio stream of the master device itself and / or the audio stream of the at least one slave device;

[0011] At least one audio stream is added to the spliced ​​video stream to obtain a composite video stream to be displayed.

[0012] Furthermore, the step of splicing multiple video streams to obtain a spliced ​​video stream includes:

[0013] According to the predetermined sampling frequency, the frame images at the same moment are obtained from multiple video streams;

[0014] Determine the brightness parameters of each frame and the contrast parameters of each frame;

[0015] Adjusting the brightness of each frame according to the brightness parameters of each frame, and adjusting the contrast of each frame according to the contrast parameters of each frame;

[0016] The adjusted frames at the same moment are spliced ​​together to obtain a spliced ​​video stream.

[0017] Furthermore, the step of adjusting the brightness of each frame according to the brightness parameter of each frame, and adjusting the contrast of each frame according to the contrast parameter of each frame includes:

[0018] Determine the brightness reference value based on the brightness parameters of all frames corresponding to the same moment; determine the contrast reference value based on the contrast parameters of all frames corresponding to the same moment;

[0019] Adjust the brightness of each frame according to the difference between the brightness parameter of each frame and the brightness reference value;

[0020] The contrast of each frame is adjusted according to the difference between the contrast parameter of each frame and the contrast reference value.

[0021] Furthermore, the step of splicing multiple video streams to obtain a spliced ​​video stream includes:

[0022] Determine the display window corresponding to each device;

[0023] Obtaining a device identifier corresponding to each video stream, and determining the device to which each video stream belongs based on the device identifier, so as to determine a display window corresponding to each video stream;

[0024] According to the relative position of the display window of each video stream in the entire display screen, the splicing order is determined, and the multiple video streams are spliced ​​according to the splicing order to obtain the spliced ​​video stream.

[0025] Furthermore, determining the display window corresponding to each device includes:

[0026] Divide the display screen into multiple display windows according to the display requirements, and assign each display window to a corresponding device;

[0027] and / or,

[0028] The role configuration of the device to which each video stream belongs is obtained from each video stream, and corresponding display windows are allocated to the corresponding devices according to the differences in the role configurations. The role configurations include the speaker and the non-speaker. The display window corresponding to the speaker is located in the middle of the display screen and / or the size of the display window corresponding to the speaker is larger than the size of the display window corresponding to the non-speaker.

[0029] Furthermore, the step of splicing multiple video streams to obtain a spliced ​​video stream includes:

[0030] Determine whether multiple portraits are obtained from each video stream;

[0031] If multiple portraits are obtained, the multiple portraits are extracted from the video stream, and the multiple portraits are spliced ​​in the video stream to obtain an intermediate video stream;

[0032] If only one portrait is obtained, the video stream remains unchanged;

[0033] The intermediate video stream is spliced ​​with other unchanged video streams to obtain a spliced ​​video stream.

[0034] Furthermore, the adding of at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed includes:

[0035] Get the timestamp of each audio stream;

[0036] According to the timestamps of each audio stream and the timestamp of the spliced ​​video stream, the obtained audio stream is added to the corresponding position of the spliced ​​video stream to obtain a composite video stream to be displayed, wherein the timestamps of the audio stream are aligned with the timestamp of the spliced ​​video stream.

[0037] Furthermore, the step of adding the acquired audio stream to the corresponding position of the spliced ​​video stream according to the timestamps of the respective audio streams and the timestamp of the spliced ​​video stream to obtain the composite video stream to be displayed includes:

[0038] If there are at least two audio streams from different devices at the same time, merge the at least two audio streams to obtain a merged audio stream;

[0039] According to the timestamp of the fused audio stream and the timestamp of the spliced ​​video stream, the fused audio stream is added to the corresponding position of the spliced ​​video stream to obtain the composite video stream to be displayed.

[0040] Furthermore, the adding of at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed includes:

[0041] Adding at least one audio stream to the spliced ​​video stream according to a timestamp;

[0042] Obtaining a target device to which at least one audio stream belongs, and determining a video stream corresponding to the target device from the spliced ​​video stream;

[0043] Using the target device as a medium, at least one audio stream is associated with a corresponding video stream, so that when the audio stream is played, the video stream corresponding to the currently played audio stream is displayed differentially with respect to other video streams.

[0044] Furthermore, the data transmission method further includes:

[0045] Sending the synthesized video stream to the interactive smart tablet so that the interactive smart tablet can display the synthesized video stream;

[0046] Receive a feedback message from the interactive smart tablet, and selectively control at least one of a data transmission rate, a video resolution, or data retransmission based on the feedback message.

[0047] To solve the above technical problems, in a third aspect, an embodiment of the present invention provides an audio and video device, comprising: an image acquisition unit, an audio unit, a communication unit, and a processing unit;

[0048] The image acquisition unit is used to obtain the video stream on the device side;

[0049] The audio unit is used to obtain the audio stream on the device side;

[0050] The communication unit is used to obtain a video stream from at least one slave device and an audio stream from at least one slave device;

[0051] The processing unit is configured to obtain a plurality of video streams consisting of a video stream of the master device itself and a video stream of at least one slave device, and splice the plurality of video streams to obtain a spliced ​​video stream; and obtain at least one audio stream consisting of an audio stream of the master device itself and / or an audio stream of at least one slave device, and add the at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed;

[0052] The communication unit is further configured to send the synthesized video stream to the interactive smart tablet.

[0053] To solve the above technical problems, in a third aspect, an embodiment of the present invention provides an audio and video device, including:

[0054] at least one processor; and,

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-device data transmission method described in the first aspect.

[0057] To solve the above technical problems, in the fourth aspect, an embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores computer-executable instructions, which are executed by one or more processors to complete the multi-device data transmission method described in the first aspect.

[0058] Compared with the prior art, the beneficial effects of the present invention are: by establishing a joint collaborative transmission channel between multiple audio and video devices, rapid sharing of audio data and video data is achieved, greatly improving the efficiency and quality of conference communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] One or more embodiments are exemplarily described by pictures in the corresponding drawings. These exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise stated, the figures in the drawings do not constitute a scale limitation.

[0060] Figure 1 This is a structural diagram of a master device, a slave device, and an interactive smart tablet provided by an embodiment of the present invention;

[0061] Figure 2 This is a structural diagram of another master device, slave device, and interactive smart tablet provided by an embodiment of the present invention;

[0062] Figure 3 This is a flowchart of a multi-device data transmission method provided by an embodiment of the present invention;

[0063] Figure 4 The embodiment of the present invention provides Figure 3 The first flow chart of step 11 in FIG.

[0064] Figure 5 The embodiment of the present invention provides Figure 3 The second flow diagram of step 11 in FIG.

[0065] Figure 6 This is a schematic diagram of a screen display effect provided by an embodiment of the present invention;

[0066] Figure 7 This is another schematic diagram of a screen display effect provided by an embodiment of the present invention;

[0067] Figure 8This is another schematic diagram of a screen display effect provided by an embodiment of the present invention;

[0068] Figure 9 The embodiment of the present invention provides Figure 3 A third flow diagram of step 11 in FIG.

[0069] Figure 10 The embodiment of the present invention provides Figure 3 The first flow diagram of step 13 in FIG.

[0070] Figure 11 The embodiment of the present invention provides Figure 3 The second flow diagram of step 13 in FIG.

[0071] Figure 12 This is another schematic diagram of a screen display effect provided by an embodiment of the present invention;

[0072] Figure 13 This is a structural diagram of an audio and video device provided by an embodiment of the present invention;

[0073] Figure 14 is a structural diagram of another audio and video device provided by an embodiment of the present invention;

[0074] Figure 15 This is a structural diagram of another audio and video device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, without departing from the scope of the present invention, a number of variations and improvements may be made by those skilled in the art. These all fall within the scope of protection of the present invention.

[0076] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0077] It should be noted that, unless there is a conflict, the various features of the embodiments of the present invention may be combined with each other and are all within the scope of protection of this application. In addition, although the functional modules are divided in the device schematics and the logical order is shown in the flow charts, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow charts.

[0078] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention belongs. The terms used in this specification are intended solely for the purpose of describing specific embodiments and are not intended to limit the invention. Furthermore, the technical features described below in the various embodiments of the invention may be combined with one another as long as they do not conflict with one another.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0080] Before introducing the solution of the embodiment of the present invention, the application scenario of the embodiment is described first. Figure 1 As shown, multiple devices are deployed in a conference room, teaching place or training place, each of which includes an image acquisition unit and an audio unit. The image acquisition unit is used to obtain portrait information, and the audio unit is used to obtain sound information.

[0081] In actual application scenarios, one device is selected as the master device based on the deployment of each device, and the other devices are selected as slave devices. Specifically, the device connected to the interactive intelligent panel (IIP) is selected as the master device; or the device connected to the interactive intelligent panel via a Wi-Fi dongle is selected as the master device. After the master device is selected, it broadcasts to the other slave devices, so that the slave devices can send the collected audio and video data to the master device.

[0082] The main device can be connected to the interactive smart tablet via a wired or wireless method, wherein the interactive smart tablet can be a control terminal such as a personal PC (Personal Computer) or a conference board. In the case of a wired method, the main device is connected to the interactive smart tablet via a USB (Universal Serial Bus), and the communication between the main device and the interactive smart tablet is achieved through the USB standard UAC (USB Audio Class) audio transmission protocol and HID control protocol; optionally, the main device can also use an Ethernet connection. In the case of a wireless method, two common wireless communication methods, wifi or Bluetooth, can be used for connection, and then a custom data frame protocol (the specific protocol analysis will be introduced later) is used for data transmission.

[0083] Similarly, the master device and the slave device are also connected wirelessly or wired.

[0084] In actual application scenarios, before establishing a wireless transmission link with the interactive smart tablet, the interactive smart tablet can optionally send a pairing request to remind the interactive smart tablet to access and prompt the user whether to enable the main device.

[0085] The master device receives data from other slave devices, then combines its own data with the data from the slave devices and sends the combined data to the interactive smart tablet. This data transmission process will be described in detail later.

[0086] In actual application scenarios, such as Figure 2 As shown, the image acquisition unit may be a camera, wherein the camera is a visible light camera and / or an infrared camera, and the audio unit may be a microphone. Specifically, the aforementioned device may be an audio and video device.

[0087] The main problem solved by the embodiments of the present invention is how to realize the joint transmission and integrated processing of audio and video data in multiple audio and video devices with cameras and microphones. The core of the present invention is that there is a master device and several slave devices in multiple audio and video devices, and the slave devices transmit their own audio and video data to the master device via WIFI. After the master device receives the data from the slave devices, it can splice the camera images of multiple slave devices into one image, and add the recorded audio at the corresponding time to integrate into a video stream with sound and picture. The master device compresses and encodes the processed video stream and sends it to the interactive smart tablet. The data transmission process of multiple devices is described in detail below with reference to the following embodiments.

[0088] Example 1:

[0089] To solve the above problems, this embodiment provides a multi-device data transmission method. Figure 3 ,include:

[0090] Step 10: The master device obtains a video stream from at least one slave device to obtain a multi-channel video stream consisting of the video stream of the master device itself and the video stream of at least one slave device.

[0091] In this embodiment, both the master device and the slave device are audio and video devices, and the master device and the slave device are deployed in a predetermined space, where the predetermined space may be a conference room, a classroom, or a training venue.

[0092] In a preferred embodiment, the master device and the slave device communicate via WIFI. This wireless communication method facilitates dynamic deployment or adjustment of the device's location and is more scalable. Similarly, the master device also preferably communicates with the interactive smart tablet via WIFI.

[0093] This section describes the use of various devices in a conference scenario. A master device and multiple slave devices are placed on a conference table. Each device captures video information from participants within its applicable range. The master device also receives video streams from slave devices, generating a multi-channel video stream consisting of its own video stream and at least one slave device's video stream.

[0094] Step 11: Splice multiple video streams to obtain a spliced ​​video stream.

[0095] The main device splices multiple video streams to obtain a spliced ​​video stream, so as to send the video streams of all participants to the interactive smart tablet and display them on the interactive smart tablet.

[0096] The interactive smart tablet can also be connected to display terminals at other locations through a network, and display the data it receives on the display terminals at other locations to realize remote conferencing.

[0097] Step 12: The master device obtains the audio stream from at least one slave device to obtain at least one audio stream consisting of the master device's own audio stream and / or the audio stream of at least one slave device.

[0098] In this embodiment, audio information from different persons may be received at the same time, audio information from only one person may be received at the same time, or audio information from no person may not be received at the same time.

[0099] After collecting the audio information, the slave device filters and performs noise reduction processing on the audio information, and then sends the processed audio information to the master device.

[0100] If audio information from different persons is received at the same time, the audio information from different persons will be superimposed and fused.

[0101] This example illustrates the use of various devices in a conference scenario. A master device and multiple slave devices are placed on a conference table. Each device collects audio information from participants within its applicable range. The master device also receives audio streams from slave devices, generating at least one audio stream composed of the master device's own audio stream and at least one slave device's audio stream.

[0102] Step 13: Add at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed.

[0103] In this embodiment, at least one audio stream is added to the spliced ​​video stream according to the timestamp to obtain a composite video stream to be displayed. In this way, the audio and video from multiple devices are merged to obtain a composite video stream with both sound and picture.

[0104] For example, in a meeting scenario, people who are not on site can watch the composite video stream to learn about the meeting situation. In a training scenario, people who are not on site can watch the composite video stream to learn about the training situation.

[0105] In specific application scenarios, when the master device is splicing its own video stream with the video stream of the slave device, it is necessary to detect and adjust specific problems that arise during the splicing process to ensure the stability and consistency of the splicing effect. Specific problems include contrast and brightness inconsistencies. Figure 4 In step 11, it specifically includes:

[0106] Step 111a: Obtain frames at the same moment from multiple video streams according to a predetermined sampling frequency.

[0107] The predetermined frequency of adoption may be adaptively selected based on actual conditions and is not specifically limited here.

[0108] In the process of splicing video streams from different devices, in order to ensure the synchronization of the pictures, the frames at the same moment are obtained from multiple video streams, where one video stream includes multiple frames, and the frames at the same moment are grouped together.

[0109] Step 112a: Determine the brightness parameter of each frame and the contrast parameter of each frame.

[0110] In order to ensure the consistency of contrast and brightness, it is necessary to determine the brightness parameters of each frame and the contrast parameters of each frame respectively, so as to adjust the brightness of each frame according to the brightness parameters to ensure the brightness consistency of the frame; and to adjust the contrast of each frame according to the contrast parameters to ensure the contrast consistency of the frame.

[0111] Step 113a: Adjust the brightness of each frame according to the brightness parameter of each frame, and adjust the contrast of each frame according to the contrast parameter of each frame.

[0112] Specifically, a brightness reference value is determined based on the brightness parameters of all frames corresponding to the same moment; a contrast reference value is determined based on the contrast parameters of all frames corresponding to the same moment; the brightness of each frame is adjusted based on the difference between the brightness parameter of each frame and the brightness reference value; and the contrast of each frame is adjusted based on the difference between the contrast parameter of each frame and the contrast reference value.

[0113] The brightness reference value and the contrast reference value are both medians or set according to weights. Of course, a brightness value that is more consistent with human perception can be selected as the brightness reference value, or a contrast value that is more consistent with human perception can be selected as the contrast reference value.

[0114] Step 114a: Splice the adjusted frames at the same moment to obtain a spliced ​​video stream.

[0115] The contrast-adjusted and brightness-adjusted frames are spliced ​​together according to the time corresponding to the frames to obtain a spliced ​​video stream.

[0116] The above is about how to adjust the video stream to explain how to splice. The following is about how to splice from the perspective of the screen display format. Figure 5 In step 11, it specifically includes:

[0117] Step 111b: Determine the display window corresponding to each device.

[0118] In an optional embodiment, the display screen is divided into multiple display windows according to screen display requirements, and each display window is allocated to a corresponding device.

[0119] The screen display requirements may be pre-set or dynamically set based on user requirements and / or the number of video streams.

[0120] like Figure 6 As shown, the screen display requirement is to display three display windows per row. The display screen is divided into multiple display windows according to the number of video streams (the sum of the number of master devices and slave devices) and the number of display windows per row, and each display window is allocated to a corresponding device.

[0121] That is, in Figure 6 In the display window, each small box corresponds to a display window, and each display window displays the video stream of one device. For example, video stream 1 is the video stream corresponding to device 1, video stream 2 is the video stream corresponding to device 2, and so on. Video stream n is the video stream corresponding to device n. The aforementioned devices are either the master device or one of the slave devices.

[0122] In another optional embodiment, the role configuration of the device to which each video stream belongs is obtained, and the corresponding display window is allocated to the corresponding device according to the difference in the role configuration, wherein the role configuration includes the speaker and the non-speaker, and the display window corresponding to the speaker is in the middle position of the display screen and / or the size of the display window corresponding to the speaker is larger than the size of the display window corresponding to the non-speaker.

[0123] The role configuration refers to the role of the person in the meeting, training, or education that the device is designed for. Specifically, it can be the presenter or non-presenter.

[0124] like Figure 7 As shown, the role corresponding to the video stream 6 is configured as the speaker, and the corresponding display window is in the middle of the display screen, and the size of the display window corresponding to the speaker is larger than the size of the display window corresponding to the non-speaker.

[0125] Furthermore, for the main speaker, the corresponding video stream needs to be placed in a more prominent position and / or larger in size. For non-main speakers, they can also be subdivided and the corresponding display windows can be allocated according to the importance of the role configuration. The more important the role, the more conspicuous the position and / or the larger the size.

[0126] Specifically, each role configuration has a corresponding priority. The higher the priority, the higher the importance. The corresponding display window can be allocated to the corresponding device according to the difference in priority of the role configuration.

[0127] like Figure 8 As shown, the role configuration corresponding to video stream 6 has the highest priority, and the role configuration corresponding to video stream 5 and video stream 7 has the second highest priority. The role configuration priorities of other video streams are equal, then the display window corresponding to video stream 6 is in the middle of the display screen and has the largest size; the display windows corresponding to video stream 5 and video stream 7 are close to the display window of video stream 6 and are slightly smaller than the display window of video stream 6. The positions of the display windows of other video streams are arranged arbitrarily and are not specifically limited here.

[0128] The display methods presented in the aforementioned embodiments can be combined with each other, and the specific methods can be determined according to actual conditions.

[0129] Step 112b: Obtain the device identification corresponding to each video stream, determine the device to which each video stream belongs based on the device identification, and determine the display window corresponding to each video stream.

[0130] Step 113b: Determine a splicing order according to the relative position of the display window of each video stream in the entire display screen, and splice the multiple video streams according to the splicing order to obtain a spliced ​​video stream.

[0131] In this embodiment, after the display windows are divided, an association is established between the display windows and the device identifiers, so that the video stream is displayed on the pre-assigned display windows.

[0132] In actual application scenarios, different portraits may be captured within the capture range of the same device. In this case, close-up images of different portraits need to be obtained from the video stream, stitched together, and then displayed. Figure 9 In step 11, it specifically includes:

[0133] Step 111c: Determine whether multiple portraits are obtained from each video stream.

[0134] Analyze each video stream to determine whether there are multiple portraits in the video stream. If multiple portraits are obtained, execute step 112c; if only one portrait is obtained, execute step 112c

[0135] Step 112c: If multiple portraits are obtained, extract the multiple portraits from the video stream, and splice the multiple portraits in the video stream to obtain an intermediate video stream.

[0136] In this embodiment, the extracted multiple portraits are spliced ​​together to obtain an intermediate video stream, so that each portrait can be given a close-up to avoid unclear visibility of some portraits.

[0137] Step 113c: If only one portrait is obtained, the video stream remains unchanged.

[0138] Step 114c: Splice the intermediate video stream with other unchanged video streams to obtain a spliced ​​video stream.

[0139] The above mainly describes the splicing of video streams. The following describes how to insert an audio stream into a corresponding video stream. In this embodiment, in order to ensure the synchronization of the audio stream and the video stream, the audio stream needs to be inserted according to the timestamp. Figure 10 In step 13, it specifically includes:

[0140] Step 131a: Obtain the timestamp of each audio stream.

[0141] In this embodiment, each audio stream carries a timestamp, each video stream also carries a timestamp, and accordingly, the spliced ​​video stream also carries a timestamp.

[0142] In this embodiment, the audio stream is added to the spliced ​​video stream according to the timestamp to obtain a composite video stream to be displayed.

[0143] Step 132a: According to the timestamps of the respective audio streams and the timestamp of the spliced ​​video stream, the acquired audio stream is added to the corresponding position of the spliced ​​video stream to obtain a composite video stream to be displayed, wherein the timestamps of the audio streams are aligned with the timestamp of the spliced ​​video stream.

[0144] In actual application scenarios, if audio information from different people is received at the same time, it is necessary to superimpose and merge the audio information of different people so that the voice information of different people can be played simultaneously. That is, if there are at least two audio streams from different devices at the same time, at least two audio streams are merged to obtain a merged audio stream. Based on the timestamp of the merged audio stream and the timestamp of the spliced ​​video stream, the merged audio stream is added to the corresponding position of the spliced ​​video stream to obtain the composite video stream to be displayed.

[0145] In actual application scenarios, in order to easily know who is sending the currently playing audio stream, in a preferred embodiment, when the audio stream is played, the corresponding display window will be displayed differently. Figure 11 In step 13, it specifically includes:

[0146] Step 131b: Add at least one audio stream to the spliced ​​video stream according to the timestamp.

[0147] Step 132b: Obtain the target device to which at least one audio stream belongs, and determine the video stream corresponding to the target device from the spliced ​​video stream.

[0148] The audio stream carries a device identifier, so the target device to which the audio stream belongs can be obtained by the device identifier. The video stream also carries a device identifier, so the video stream corresponding to the target device can be determined from the spliced ​​video stream based on the device identifier.

[0149] Step 133b: Using the target device as a medium, establish an association between at least one audio stream and a corresponding video stream, so that when playing the audio stream, the video stream corresponding to the currently played audio stream is displayed differently from other video streams.

[0150] The differentiated display includes highlighting, enlarging, shaking and / or displaying the video stream corresponding to the currently playing audio stream in a picture-in-picture display window. Figure 12 , the video stream corresponding to the currently played audio stream is video stream 4, so the display window corresponding to video stream 4 is highlighted.

[0151] In this way, you can know who is sending the currently playing audio stream, which enhances the experience.

[0152] In actual application scenarios, the smoothness of data transmission and the packet loss rate also need to be considered. In a preferred embodiment, the data transmission method also includes: sending the composite video stream to the interactive smart tablet so that the interactive smart tablet can display the composite video stream; receiving feedback messages from the interactive smart tablet, and selectively controlling at least one of the data transmission rate, video resolution or data retransmission based on the feedback messages.

[0153] Specifically, the master device compresses and encodes the composite video stream and sends it to the interactive smart tablet. The interactive smart tablet verifies and decodes the composite video stream before displaying it. When the master device sends the processed and compressed composite video stream to the interactive smart tablet, it can incorporate error coding, retransmission control, and encryption protection to ensure data reliability and security.

[0154] When the interactive smart tablet decodes the received composite video stream, it combines the buffer and flow control mechanism to ensure the decoding efficiency of the composite video stream, and ensures the smooth playback and real-time response of the composite video stream.

[0155] Specifically, the data transmission rate of the master device is controlled based on bandwidth limitations, network congestion, and flow control strategies. A feedback mechanism is designed to support interactive smart tablets in controlling the data transmission rate within the data protocol control framework. For example, if the data cannot be processed and bandwidth needs to be limited, a message can be sent to the master device to lower the video resolution. Data caching and retransmission mechanisms are provided to ensure reliable data transmission. For example, if a data frame is lost, the previously received data is first buffered, and an ACK (Acknowledgement character) is sent to the master device for retransmission. After the lost data is received, the data is read from the buffer for parsing.

[0156] In actual application scenarios, to meet the needs and scenarios of different users, multiple screen displays, multiple video stream splicing methods, and multiple viewing angles can be provided, such as picture-in-picture and split-screen. User interaction, voting, and question-and-answer functions can also be provided. For example, for interactive functions, the interactive smart tablet receives user messages, selects the device to interact with, and disables the microphones of other devices to enable interaction.

[0157] The multi-device data transmission method according to the embodiment of the present invention has the following effects:

[0158] 1. Establish communication connections between multiple audio and video devices, and realize the joint transmission of audio and video data.

[0159] 2. It realizes the stitching of multiple camera images and integrates multiple audio recording data, which improves the readability and visibility of audio and video data.

[0160] 3. Use a variety of processing technologies to make data processing faster and more accurate, so as to better meet user needs.

[0161] Example 2:

[0162] Here, we take the communication between the master device and the interactive smart tablet via WIFI, and the communication between the master device and the slave device via WIFI as examples to illustrate the specific implementation process of the WIFI wireless communication data protocol.

[0163] The data protocols of WIFI wireless communication are divided into:

[0164] Frame header: used to identify and detect data frames.

[0165] Sender: defines which slave device sends the data frame, and can carry the device identifier.

[0166] Data length: includes the length from the serial number to the check digit.

[0167] Frame sequence number: Each time a new communication is initiated, it is incremented by 1. If it exceeds 255, it is reset to 0 and counted again.

[0168] ACK: Check whether the data frame is received successfully.

[0169] Message type: such as status, control, data and other major types.

[0170] Subtype: Subtypes under each major type; for example, the status type can be divided into mute, volume, etc.

[0171] Data type: 1 byte. This refers to the data type of the payload. It can be JSON, a string, an integer, etc.

[0172] Payload: The data carried by the message, including a timestamp.

[0173] Checksum: CRC8, checksum from frame number to payload.

[0174] Example 3:

[0175] like Figure 13As shown, based on the above-mentioned embodiment, this embodiment provides an audio and video device, including: an image acquisition unit, an audio unit, a communication unit and a processing unit; the image acquisition unit is used to obtain the video stream on the device side; the audio unit is used to obtain the audio stream on the device side; the communication unit is used to obtain the video stream from at least one slave device, and the audio stream of at least one slave device; the processing unit is used to obtain multiple video streams consisting of the video stream of the main device itself and the video stream of at least one slave device, and splice the multiple video streams to obtain a spliced ​​video stream; and obtain at least one audio stream consisting of the audio stream of the main device itself and / or the audio stream of at least one slave device, add at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed; the communication unit is also used to send the composite video stream to the interactive smart tablet.

[0176] Furthermore, the processing unit is used to obtain frame images at the same moment from multiple video streams according to a predetermined sampling frequency; determine the brightness parameters and contrast parameters of each frame image; adjust the brightness of each frame image according to the brightness parameters of each frame image, and adjust the contrast of each frame image according to the contrast parameters of each frame image; and splice the adjusted frame images at the same moment to obtain a spliced ​​video stream.

[0177] The specific implementation method can be found in the above embodiments and will not be described in detail here.

[0178] Example 4:

[0179] Based on the above-mentioned embodiment 1-embodiment 3, as Figure 14 As shown, this embodiment also provides an audio and video device, including: at least one processor 21; and a memory 22 communicatively connected to the at least one processor 21; wherein the memory 22 stores instructions that can be executed by the at least one processor 21, and the instructions are executed by the at least one processor 21 so that the at least one processor 21 can execute the multi-device data transmission method described in the aforementioned embodiment.

[0180] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 14 The bus connection is taken as an example.

[0181] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multi-device data transmission method in the embodiment of the present application. The processor 21 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the multi-device data transmission method in the above method embodiment. The multi-device data transmission method includes a master device obtaining a video stream from at least one slave device to obtain a multi-channel video stream composed of the master device's own video stream and the at least one slave device's video stream; splicing the multi-channel video stream to obtain a spliced ​​video stream; the master device obtains an audio stream from at least one slave device to obtain at least one audio stream composed of the master device's own audio stream and / or the at least one slave device's audio stream; and adding at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed.

[0182] In an optional embodiment, the adding of at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed includes: adding at least one audio stream to the spliced ​​video stream according to a timestamp; obtaining a target device to which at least one audio stream belongs, and determining the video stream corresponding to the target device from the spliced ​​video stream; and associating at least one audio stream with a corresponding video stream using the target device as a medium, so that when the audio stream is played, the video stream corresponding to the currently played audio stream is displayed differently from other video streams.

[0183] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the audio and video equipment, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and these remote memories may be connected to the audio and video equipment via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The one or more modules are stored in the memory 22, and when executed by the one or more processors 21, execute the multi-device data transmission method in any of the above method embodiments, for example, execute the method steps of the multi-device data transmission method described above.

[0185] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0186] Furthermore, if Figure 15 As shown, the audio and video device may also include: Wi-Fi device 23, display screen 24, Bluetooth device 25, audio unit 26, power supply system 27, peripheral interface 28, sensor module 29, data conversion module 30 and other components. These components can communicate through one or more communication buses or signal lines. It will be understood by those skilled in the art that Figure 15 The hardware structure shown in the figure does not constitute a limitation on the audio and video device. The audio and video device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0187] The processor 21 is the control center of the audio and video equipment. It connects the various components of the audio and video equipment using various interfaces and circuits. It executes various functions of the audio and video equipment and processes data by running or executing applications stored in the memory 22 and invoking data and instructions stored in the memory 22. In some embodiments, the processor 21 may include one or more processing units. The processor 21 may also integrate an application processor 21 and a modem processor 21. The application processor 21 primarily handles the operating system, user interface, and application programs, while the modem processor 21 primarily handles wireless communications. It is understood that the modem processor 21 may not be integrated into the processor 21.

[0188] In some other embodiments of the present application, the processor 21 may further include an AI chip. The learning and processing capabilities of the AI ​​chip include image understanding capabilities, natural language understanding capabilities, and speech recognition capabilities. The AI ​​chip can enable audio and video equipment to have better performance, longer battery life, and better security and privacy. For example, if the audio and video equipment processes data through the cloud, the data needs to be uploaded for processing before the results are returned, which is very inefficient under existing technical conditions. If the local end of the audio and video equipment has strong AI learning capabilities, then the audio and video equipment does not need to upload the data to the cloud, and can be processed directly on the local end, thereby improving the security and privacy of the data while improving processing efficiency.

[0189] The memory 22 is used to store applications and data. The processor 21 executes various functions and data processing of the audio and video device by running the applications and data stored in the memory 2212. The memory 22 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and applications required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created when using the audio and video device (such as audio data, video data, etc.). In addition, the memory 22 can include a high-speed random access memory 22, and can also include a non-volatile memory 22, such as a disk memory 22, a flash memory device or other non-volatile solid-state memory 22. The memory 22 can store various operating systems, such as an operating system developed by Apple, an operating system developed by Microsoft, etc.

[0190] The display screen 24 is used to display images, videos, etc. The display screen 24 can be a touch screen. In some embodiments, the audio and video device may include one or N display screens 24, where N is a positive integer greater than 1. The audio and video device implements its display function through the GPU, the display screen 24, and the application processor 21. The GPU is a microprocessor 21 for image processing, connecting the display screen 24 and the application processor 21. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 21 may include one or more GPUs, which execute program instructions to generate or change display information.

[0191] The Wi-Fi device 23 is used to provide network access that complies with Wi-Fi-related standard protocols for audio and video equipment. Audio and video equipment can access the Wi-Fi access point through the Wi-Fi device 23, thereby helping to browse web pages and access streaming media, etc., which provides users with wireless broadband Internet access. Audio and video equipment can also establish a Wi-Fi connection with a terminal device connected to the Wi-Fi access point through the Wi-Fi device 23 and the Wi-Fi access point for mutual data transmission. In some other embodiments, the Wi-Fi device 23 can also serve as a Wi-Fi wireless access point, which can provide Wi-Fi network access for other electronic devices. For example, the Wi-Fi device 23 includes a first wireless network card, a second wireless network card and a third wireless network card.

[0192] The Bluetooth device 25 is used to implement data exchange between the audio and video device and other short-range electronic devices (such as terminals, smart watches, etc.). The Bluetooth device 25 in the embodiment of the present application can be an integrated circuit or a Bluetooth chip.

[0193] Audio unit 26, speakers, and microphones provide an audio interface between the user and audio and video equipment. Audio unit 26 converts received audio data into electrical signals and transmits them to the speakers, which then convert them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by audio unit 26 and converted into audio data. The audio data is then transmitted to the terminal via the internet, Wi-Fi, or Bluetooth, or is output to memory 22 for further processing.

[0194] The power supply system 27 is used to charge the various components of the audio and video equipment. The power supply system 27 can include a battery and a power management module. The battery can be logically connected to the processor 21 through a power management chip, so that the power supply system 27 can manage charging, discharging, and power consumption.

[0195] The peripheral interface 28 is used to provide various interfaces for external input / output devices (such as an external display, external memory 22, and a subscriber identity module card). For example, the external memory 22 interface can be connected to an external memory 22, such as a MicroSD card, to expand the storage capacity of audio and video devices. The peripheral interface 28 can be used to couple these external input / output peripheral devices to the processor 21 and memory 22.

[0196] The sensor module 29 may include at least one sensor. For example, a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor. Among them, the ambient light sensor can adjust the brightness of the display screen 24 according to the brightness of the ambient light. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of audio and video equipment (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), etc. Of course, according to actual needs, the sensor module 29 can also include any other feasible sensors.

[0197] The data conversion module 30 may include a digital-to-analog converter and an analog-to-digital converter. The functions of the digital-to-analog converter and the analog-to-digital converter may be explained in the corresponding technical terms mentioned above and will not be repeated here.

[0198] An embodiment of the present application also provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the method steps of the multi-device data transmission method described above. The multi-device data transmission method includes a master device acquiring a video stream from at least one slave device to obtain a multi-channel video stream composed of the master device's own video stream and the at least one slave device's video stream; splicing the multi-channel video stream to obtain a spliced ​​video stream; the master device acquiring an audio stream from at least one slave device to obtain at least one audio stream composed of the master device's own audio stream and / or the at least one slave device's audio stream; and adding the at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed.

[0199] An embodiment of the present application also provides a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the multi-device data transmission method in any of the above method embodiments, for example, executing the method steps of the multi-device data transmission method described above.

[0200] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0201] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. For the sake of simplicity, they are not provided in detail. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-device data transmission method, characterized in that: include: The master device acquires a video stream from at least one slave device to acquire a multi-channel video stream consisting of the video stream of the master device itself and the video stream of at least one slave device; Splicing multiple video streams to obtain a spliced ​​video stream; The master device acquires the audio stream from at least one slave device to acquire at least one audio stream composed of the audio stream of the master device itself and / or the audio stream of at least one slave device; At least one audio stream is added to the spliced ​​video stream to obtain a composite video stream to be displayed.

2. The multi-device data transmission method according to claim 1, characterized in that: The step of splicing multiple video streams to obtain a spliced ​​video stream includes: According to a predetermined sampling frequency, frames at the same moment are obtained from multiple video streams; Determine the brightness parameters of each frame and the contrast parameters of each frame; The brightness of each frame is adjusted according to the brightness parameter of each frame, and the contrast of each frame is adjusted according to the contrast parameter of each frame; The adjusted frame images at the same time are spliced ​​together to obtain a spliced ​​video stream.

3. The multi-device data transmission method according to claim 2, characterized in that: The step of adjusting the brightness of each frame according to the brightness parameter of each frame, and adjusting the contrast of each frame according to the contrast parameter of each frame includes: Determine the brightness reference value according to the brightness parameters of all the frames corresponding to the same moment; determine the contrast reference value according to the contrast parameters of all the frames corresponding to the same moment; According to the difference between the brightness parameter of each frame and the brightness reference value, the brightness of each frame is adjusted; According to the difference between the contrast parameter of each frame and the contrast reference value, the contrast of each frame is adjusted.

4. The multi-device data transmission method according to claim 1, characterized in that: The step of splicing multiple video streams to obtain a spliced ​​video stream includes: Determine the display window corresponding to each device; Obtaining a device identifier corresponding to each video stream, and determining the device to which each video stream belongs according to the device identifier, so as to determine a display window corresponding to each video stream; According to the relative position of the display window of each video stream in the entire display screen, the splicing order is determined, and the multiple video streams are spliced ​​according to the splicing order to obtain the spliced ​​video stream.

5. The multi-device data transmission method according to claim 4, characterized in that: Determining the display window corresponding to each device includes: Divide the display screen into multiple display windows according to the screen display requirements, and allocate each display window to a corresponding device; and / or, The role configuration of the device to which each video stream belongs is obtained from each video stream, and a corresponding display window is allocated to the corresponding device according to the difference in the role configuration, wherein the role configuration includes a speaker and a non-speaker, and the display window corresponding to the speaker is located in the middle of the display screen and / or the size of the display window corresponding to the speaker is larger than the size of the display window corresponding to the non-speaker.

6. The multi-device data transmission method according to any one of claims 1 to 5, characterized in that: The step of splicing multiple video streams to obtain a spliced ​​video stream includes: Determine whether multiple portraits are obtained from each video stream; If multiple portraits are obtained, the multiple portraits are extracted from the video stream, and the multiple portraits are spliced ​​in the video stream to obtain an intermediate video stream; If only one portrait is obtained, the video stream remains unchanged; The intermediate video stream is spliced ​​with other unchanged video streams to obtain a spliced ​​video stream.

7. The multi-device data transmission method according to any one of claims 1 to 5, characterized in that: The step of adding at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed comprises: Get the timestamp of each audio stream; According to the timestamps of each audio stream and the timestamp of the spliced ​​video stream, the acquired audio stream is added to the corresponding position of the spliced ​​video stream to obtain a composite video stream to be displayed, wherein the timestamp of the audio stream is aligned with the timestamp of the spliced ​​video stream.

8. The multi-device data transmission method according to claim 7, characterized in that: The step of adding the acquired audio stream to the corresponding position of the spliced ​​video stream according to the timestamps of the respective audio streams and the timestamp of the spliced ​​video stream to obtain the composite video stream to be displayed comprises: If there are at least two audio streams from different devices at the same time, merge the at least two audio streams to obtain a merged audio stream; According to the timestamp of the fused audio stream and the timestamp of the spliced ​​video stream, the fused audio stream is added to the corresponding position of the spliced ​​video stream to obtain a composite video stream to be displayed.

9. The multi-device data transmission method according to any one of claims 1 to 5, characterized in that: The step of adding at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed comprises: Adding at least one audio stream to the spliced ​​video stream according to a timestamp; Obtaining a target device to which at least one audio stream belongs, and determining a video stream corresponding to the target device from the spliced ​​video stream; Using the target device as a medium, at least one audio stream is associated with a corresponding video stream, so that when the audio stream is played, the video stream corresponding to the currently played audio stream is displayed differentially relative to other video streams.

10. The multi-device data transmission method according to any one of claims 1 to 5, characterized in that: The data transmission method further comprises: Sending the synthesized video stream to an interactive smart tablet so that the interactive smart tablet can display the synthesized video stream; A feedback message is received from the interactive smart tablet, and at least one of a data transmission rate, a video resolution, or data retransmission is selectively controlled based on the feedback message.

11. An audio and video device, characterized in that: include: Image acquisition unit, audio unit, communication unit and processing unit; The image acquisition unit is used to obtain the video stream on the device side; The audio unit is used to obtain the audio stream on the device side; The communication unit is used to obtain a video stream from at least one slave device and an audio stream from at least one slave device; The processing unit is used to obtain a multi-channel video stream consisting of a video stream of the master device itself and a video stream of at least one slave device, and splice the multi-channel video streams to obtain a spliced ​​video stream; and obtaining at least one audio stream composed of an audio stream of the master device itself and / or an audio stream of at least one slave device, and adding the at least one audio stream to the spliced ​​video stream to obtain a composite video stream to be displayed; The communication unit is also used to send the synthesized video stream to the interactive smart tablet.

12. An audio and video device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-device data transmission method according to any one of claims 1 to 10.

Images