Multi-view video processing method and related device
The audio data of multi-view video is processed through the audio alignment algorithm, the audio alignment results are obtained, and the video data is aligned based on the results, which solves the problems of low accuracy and high cost of multi-view video alignment, and achieves high accuracy and low-cost multi-view video alignment.
Patent Information
- Application Number
- CN202411918685.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2044-12-24
AI Technical Summary
In the prior art, there are problems with low alignment accuracy and high cost of multi-view videos.
The multiple audio data is processed through the audio alignment algorithm, the audio alignment result is obtained, and the multiple video data is aligned based on the result, and the display time stamp of each video data is modified to achieve the alignment of multi-view video.
Improves alignment accuracy of multi-view videos, reduces costs, and does not rely on time synchronization between video acquisition devices.
Smart Images

Figure CN119676484A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of visual networking technology, and in particular to a multi-view video processing method and related devices. Background Art
[0002] With the development and popularization of the visual network, multi-view live video has become one of the most popular live broadcast methods in scenarios such as live broadcast of tourist attractions, online game live broadcast, and e-commerce platform live broadcast. Among them, multi-view video live broadcast refers to a live broadcast method that uses multiple video acquisition devices to simultaneously capture audio and video from different angles, and aligns the captured multi-view videos according to the time dimension. The audience in the live broadcast room can watch the live content from different angles at the same time through the aligned multi-view videos, which can enrich the viewing experience and enhance the content visualization.
[0003] Currently, the existing technology usually realizes the alignment of multi-view videos according to the acquisition timestamp of the live video, which requires strict time synchronization between various video acquisition devices and / or the video acquisition devices need to perform additional processing on the acquired audio and video streams.
[0004] However, considering that each video acquisition device may have differences in device performance, function configuration, hardware configuration, etc., in some cases, the various video acquisition devices cannot achieve satisfactory time synchronization and / or cannot perform additional processing on the audio and video streams, thereby reducing the alignment accuracy of the multi-view video. Alternatively, it is necessary to replace the video acquisition device with a higher cost for acquisition, which has a high cost technical defect. Summary of the invention
[0005] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects of low alignment accuracy and high cost of multi-view videos in the prior art.
[0006] In a first aspect, an embodiment of the present application provides a method for processing a multi-view video, including:
[0007] Acquire a multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of the live audio and video data includes audio data and video data, N is a positive integer and N≥2;
[0008] Using an audio alignment algorithm to process the multiple channels of audio data and obtain an audio alignment result;
[0009] Aligning the multiple channels of video data based on the audio alignment result, and obtaining a video alignment result;
[0010] According to the video alignment result, the display timestamps corresponding to the video data of each channel are modified respectively, so that the video frames at the same position correspond to the same display timestamps.
[0011] In some embodiments, the audio alignment algorithm includes an audio feature extraction algorithm and an audio feature matching algorithm;
[0012] The method of using an audio alignment algorithm to process the multiple channels of audio data and obtaining an audio alignment result includes:
[0013] Using the audio feature extraction algorithm, respectively extract the audio features corresponding to each channel of the audio data;
[0014] Using the audio feature matching algorithm, the audio features corresponding to the audio data of each channel are matched, and the audio alignment result is obtained based on the matching result;
[0015] The audio alignment result includes N similar features and audio position information of each similar feature in corresponding audio data, and the N similar features belong to N channels of audio data.
[0016] In some embodiments, the video alignment result includes N positioning video frames and video position information of each positioning video frame in the corresponding video data;
[0017] The aligning the multiple channels of video data based on the audio alignment result and obtaining the video alignment result includes:
[0018] For each of the similar features, the live audio and video data to which the similar feature belongs is used as the target audio and video data, and the audio position information corresponding to the similar feature is used as the target position information. According to the audio-video correspondence relationship of the target audio and video data, the video frame corresponding to the target position information in the video data of the target audio and video data is used as a positioning video frame, and the video position information of the positioning video frame is determined;
[0019] The step of respectively modifying the display timestamps corresponding to the video data of each channel according to the video alignment result includes:
[0020] According to the server timestamp and the video position information corresponding to the N positioning video frames, the display timestamp corresponding to each channel of the video data is modified respectively.
[0021] In some embodiments, the method further comprises:
[0022] For each channel of the aligned N channels of live audio and video data, encode the live audio and video data according to a plurality of preset encoding specifications, and obtain a plurality of live encoded data; wherein the encoding specifications include resolution and frame rate;
[0023] According to the various live broadcast encoding data, live broadcast video is pushed to the live broadcast terminal.
[0024] In some embodiments, the live video is pushed to the live terminal according to the various live encoding data, including:
[0025] Based on the predetermined multicast address information, the various live broadcast encoded data are multicast transmitted to push the live broadcast video to the live broadcast terminal.
[0026] In some embodiments, the pushing of live video to the live broadcast terminal according to the various live broadcast encoding data further includes:
[0027] Receiving a live broadcast request sent by the live broadcast terminal; wherein the live broadcast request carries decoding capability information of the live broadcast terminal;
[0028] Determine a target encoding specification according to the decoding capability information, and generate request response information based on target multicast address information corresponding to the target encoding specification;
[0029] Return the request response information to the live broadcast terminal.
[0030] In some embodiments, the acquiring of multi-view video includes:
[0031] According to a plurality of preset paths, live audio and video data collected by a plurality of video collection devices are received respectively; wherein the plurality of preset paths correspond one-to-one to the plurality of channels of live audio and video data.
[0032] In a second aspect, an embodiment of the present application provides a multi-view video processing device, including:
[0033] A video acquisition module, used to acquire multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of the live audio and video data includes audio data and video data, N is a positive integer and N≥2;
[0034] An audio alignment result acquisition module is used to process the multiple channels of audio data using an audio alignment algorithm and obtain an audio alignment result;
[0035] A video alignment module, used for aligning the multiple channels of video data based on the audio alignment result, and obtaining a video alignment result;
[0036] The timestamp modification module is used to modify the display timestamps corresponding to the video data of each channel according to the video alignment result, so that the video frames at the same position correspond to the same display timestamps.
[0037] In a third aspect, an embodiment of the present application provides a storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-view video processing method described in any of the above embodiments.
[0038] In a fourth aspect, an embodiment of the present application provides a live video server, the live video server comprising: one or more processors, and a memory;
[0039] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the multi-view video processing method described in any of the above embodiments are performed.
[0040] In the multi-view video processing method and related device provided in some embodiments of the present application, the live video server can separate the respective audio data from each live audio and video data of the multi-view video, and use the audio alignment algorithm to perform audio synchronization on each audio data to achieve multi-view audio alignment. The live video server can align each channel of video data according to the time dimension based on the audio alignment result, and modify the display timestamps corresponding to each channel of video data based on the video alignment result, so that the video frames at the same position correspond to the same display timestamps. In this way, multi-view video alignment can be achieved. It can be seen that the present application does not require the video acquisition device to perform additional processing on the collected audio and video data, nor does it rely on the time synchronization between multiple video acquisition devices. Even if the video acquisition device does not meet the time synchronization requirements, the present application can accurately achieve the alignment and synchronization of multi-view videos, thereby improving the alignment accuracy of multi-view videos and reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0042] Figure 1 is one of the system schematic diagrams of a multi-view video processing system in one embodiment;
[0043] Figure 2 One of the flowcharts of a method for processing multi-view videos in one embodiment;
[0044] Figure 3 FIG2 is a second system schematic diagram of a multi-view video processing system in one embodiment;
[0045] Figure 4 1 is a schematic diagram of functional modules of a multi-view video processing system in one embodiment;
[0046] Figure 5 FIG2 is a second flow chart of a method for processing multi-view videos in one embodiment;
[0047] Figure 6 is a schematic structural diagram of a multi-view video processing device in one embodiment;
[0048] Figure 7 FIG. 4 is an internal structure diagram of a live video server in one embodiment. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0050] In some embodiments, the multi-view video processing method provided in the present application can be applied to a multi-view video processing system. It can be understood that the multi-view video processing system can be used to provide a multi-view video live broadcast service in the field of visual networking, and / or provide a multi-view video live broadcast service in other video scenarios.
[0051] Among them, live video (LV) refers to the technology of capturing live scenes through video acquisition equipment and pushing audio and video data to the network in real time through network transmission, so that users can receive and watch videos through terminals such as browsers or application APPs. Multi-view video can include N channels of live audio and video data, and any two channels of live audio and video data can correspond to different shooting angles, where N is a positive integer and N≥2.
[0052] It can be understood that the number of N can be determined according to actual conditions, and this document does not impose any specific restrictions on this. For example, it can be 2, 3, 4, 5, 8, etc. For ease of description, some embodiments of the present application are described by taking N=4 as an example, that is, the multi-view video is a 4-view video, which includes 4 channels of live audio and video data.
[0053] like Figure 1As shown, the multi-view video processing system may include a video acquisition device 102, a live video server 104 and a live terminal 106. The video acquisition device 102 refers to a device for acquiring live audio and video data, which may be, but not limited to, a video camera, a camera, etc. The live terminal 106 refers to a terminal device for playing multi-view videos, which may be, but not limited to, a smart phone, a wearable device, a home appliance, an Internet of Things device, a tablet computer, a laptop computer, a desktop computer, etc. The live video server 104 is used to align multiple channels of live audio and video data in the multi-view video according to the time dimension, so that the aligned multi-view video can show the audio and video content corresponding to different perspectives at the same time.
[0054] It is understandable that there may be multiple video acquisition devices 102, and there may be one or more live broadcast terminals 106. The specific numbers of video acquisition devices 102 and live broadcast terminals 106 may be determined according to actual conditions, and this document does not impose any specific restrictions on this.
[0055] Specifically, each video capture device 102 can capture live audio and video data of the corresponding perspective, and transmit the captured live audio and video data to the live video server 104 in real time. The live video server 104 can receive the live audio and video data uploaded by each video capture device 102 respectively, so as to obtain a multi-perspective video. The live video server 104 can also process the multi-perspective video according to the multi-perspective video processing method provided in the following embodiment to align the live audio and video data of each channel in the time dimension. In this way, the acquisition capability / processing capability of the video capture device 102 can be unchanged, and strict requirements are not made on the time synchronization of each video capture device 102. By introducing the audio alignment synchronization technology, the alignment and synchronization of the multi-perspective video is completed in the live video server 104.
[0056] The live broadcast terminal 106 may report the capability information such as the number of simultaneous decoding channels supported by the terminal, the resolution / frame rate of each channel, etc. to the live broadcast video server 104. The live broadcast video server 104 may also push the aligned multi-view video to the live broadcast terminal 106 based on the received information, and the multi-view video pushed by the live broadcast video server 104 matches the terminal decoding capability of the live broadcast terminal 106. After receiving the live broadcast audio and video data sent by the live broadcast video server 104, the live broadcast terminal 106 may decode the live broadcast audio and video data, and render the video according to the display timestamp to synchronously play the multi-view video.
[0057] The following describes the multi-view video processing method provided by this application.
[0058] In some embodiments, the present application provides a method for processing multi-view video. The following embodiments apply this method to Figure 1The live video server shown in the figure is used as an example. Figure 2 As shown, the method may include the following steps:
[0059] S202: Acquire a multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of live audio and video data includes audio data and video data, N is a positive integer and N≥2;
[0060] S204: Processing the multiple channels of audio data using an audio alignment algorithm and obtaining an audio alignment result;
[0061] S206: aligning the multiple channels of video data based on the audio alignment result, and obtaining a video alignment result;
[0062] S208: According to the video alignment result, respectively modify the display timestamps corresponding to the video data of each channel, so that the video frames at the same position correspond to the same display timestamp.
[0063] In this application, the live video server can obtain N channels of live audio and video data collected by multiple video acquisition devices. It can be understood that in multiple channels of live audio and video data, the shooting angles corresponding to each two channels of live audio and video are different, so that multiple channels of live audio and video data can present live content at different angles. Each channel of live audio and video data can include audio data and video data, and the audio data and video data in the same channel of live audio and video data have a time correspondence, so that a single channel of live audio and video data can achieve audio and video synchronization.
[0064] In the case of obtaining N channels of live audio and video data, the live video server can perform audio extraction on each channel of live audio and video data to separate N channels of audio data from the N channels of live audio and video data. For example, when N=2, the live video server can perform audio extraction on the first channel of live audio and video data and the second channel of live audio and video data, and obtain the audio data corresponding to the first channel of live audio and video data and the audio data corresponding to the second channel of live audio and video data, respectively, and add up the two channels of audio data.
[0065] When N channels of audio data are obtained, the live video server can use an audio alignment algorithm to process the N channels of audio data and obtain an audio alignment result. The audio alignment algorithm refers to an algorithm used to determine the alignment relationship of multiple channels of audio data on the time axis, and may include, for example, a time series analysis algorithm, a dynamic time warping algorithm, etc. The audio alignment result is used to describe the alignment relationship of the N channels of audio data on the time axis, for example, it may be the time difference between each channel of target audio data and the reference audio data, wherein the reference audio data is one channel of audio data in the N channels, and the target audio data is N-1 channels of audio data in the N channels of audio data except the reference audio data.
[0066] It is understandable that the present application can use any method to implement audio extraction, and the specific algorithm type and specific implementation method of the audio extraction algorithm can be determined according to actual conditions, and the present application does not impose specific restrictions on this. Similarly, the present application can use any audio alignment algorithm to obtain the audio alignment result, and the present application does not impose specific restrictions on this.
[0067] Since the audio and video data of each live audio and video data have a time correspondence, and the alignment relationship of N-channel audio data in the time dimension can be reflected by the audio alignment result, after determining the audio alignment result of N-channel audio data, the N-channel video data can be aligned in the time dimension according to the audio alignment result, and the video alignment result can be obtained. The video alignment result is used to describe the alignment relationship of N-channel video data on the time axis.
[0068] After determining the alignment relationship of N channels of video data, the present application can modify the PTS (Presentation Time Stamp) corresponding to the N channels of video data respectively according to the video alignment result, so that video frames at the same time position correspond to the same display timestamp.
[0069] For example, when N=2, if the first video frame of the first video data (hereinafter referred to as video frame 1) is time-aligned with the second video frame of the second video data (hereinafter referred to as video frame 2), that is, video frame 1 and video frame 2 correspond to the same time position, then the PTS of video frame 1 and the PTS of video frame 2 can be modified to the same timestamp, so that video frame 1 and video frame 2 correspond to the same display timestamp. The PTS corresponding to the other video frames of the first video data and the other video frames of the second video data can be modified in accordance with the above method, so that the video frames at the same time position correspond to the same display timestamp.
[0070] It can be seen that the present application does not require the video acquisition device to perform additional processing on the collected audio and video data, nor does it rely on the time synchronization between multiple video acquisition devices. Even if the video acquisition device does not meet the time synchronization requirements, the present application can accurately achieve the alignment and synchronization of multi-view videos, thereby improving the alignment accuracy of multi-view videos and reducing costs.
[0071] In some embodiments, the audio alignment algorithm includes an audio feature extraction algorithm and an audio feature matching algorithm. Among them, the audio feature extraction algorithm refers to an algorithm for extracting audio features from audio data. The audio feature matching algorithm refers to an algorithm for feature matching of audio features corresponding to multiple channels of audio data. Furthermore, the audio features described in the present application may be the result of digitally representing a certain attribute of the audio data, such as the time domain features, frequency domain features, Mel spectrum, etc. of the audio data.
[0072] The audio alignment algorithm is used to process multi-channel audio data and obtain audio alignment results, including:
[0073] Step A1: using an audio feature extraction algorithm to extract audio features corresponding to each channel of audio data;
[0074] Step A3: Using an audio feature matching algorithm, the audio features corresponding to each channel of audio data are matched, and an audio alignment result is obtained based on the matching result.
[0075] The audio alignment result includes N similar features and audio position information of each similar feature in the corresponding audio data, and the N similar features belong to N channels of audio data.
[0076] In this embodiment, the live video server can use an audio feature extraction algorithm to obtain the audio features corresponding to each channel of audio data, and use an audio feature matching algorithm to perform feature matching on the audio features corresponding to the N channels of audio data, so as to identify similar audio features possessed by the N channels of audio data, and then obtain an audio alignment result. In this way, the audio alignment result can be obtained through the audio synchronization technology based on feature recognition, thereby further improving the accuracy of the alignment.
[0077] Specifically, the live video server can use an audio feature extraction algorithm to perform feature recognition processing on N channels of audio data respectively to obtain audio features corresponding to each channel of audio data. When the audio features corresponding to the N channels of audio data are obtained, the live video server can use an audio feature matching algorithm to perform feature matching on the N channels of audio features, and obtain N similar features and audio position information of each similar feature in the corresponding audio data based on the feature matching results.
[0078] The N similar features are audio features used to reflect the same audio content in the N channels of audio data. In some examples, the N similar features can be understood as common audio features of the N channels of audio data. Further, the live video server can align according to a preset feature matching threshold so that the feature similarity between the N similar features satisfies the preset feature matching condition.
[0079] The audio position information of each similar feature in the corresponding audio data refers to the position information of the similar feature in the audio data to which it belongs. For example, when the first similar feature is the audio feature of the first audio data and the second similar feature is the audio feature of the second audio data, the audio position information corresponding to the first similar feature refers to the position information of the first similar feature in the first audio data, and the audio position information corresponding to the second similar feature refers to the position information of the second similar feature in the second audio data.
[0080] It should be noted that the position information corresponding to the similar features can be described by parameters in different forms, and this application does not impose any specific restrictions on this. It only needs to be able to describe the time position of the similar features in the corresponding audio data. For example, this application can use the audio frame number corresponding to the similar features as the position information, or use the timestamp corresponding to the similar features in the corresponding audio data as the position information.
[0081] In some embodiments, the video alignment result includes N positioning video frames and video position information of each positioning video frame in corresponding video data, wherein the N positioning video frames belong to N channels of video data.
[0082] Align multiple video data channels based on the audio alignment result, and obtain the video alignment result, including:
[0083] For each similar feature, the live audio and video data to which the similar feature belongs is used as the target audio and video data, and the audio position information corresponding to the similar feature is used as the target position information. According to the audio and video correspondence relationship of the target audio and video data, the video frame corresponding to the target position information in the video data of the target audio and video data is used as a positioning video frame, and the video position information of the positioning video frame is determined;
[0084] According to the video alignment results, modify the display timestamps corresponding to each channel of video data, including:
[0085] According to the server timestamp and the video position information corresponding to the N positioning video frames, the display timestamp corresponding to each channel of video data is modified respectively.
[0086] In this embodiment, since the N similar features of the audio alignment results are audio features used to reflect the same audio content in N channels of audio data, the live broadcast server can determine the positioning video frames corresponding to the same audio content in each channel of video data based on the correspondence between the audio and video data of the same channel and the audio position information corresponding to the N similar features, and obtain N positioning video frames and the video position information corresponding to each positioning video frame.
[0087] The following is an example of determining a positioning video frame in one channel of video data. It can be understood that the positioning video frames of N channels of video data can be determined in the following manner. Assume that one of the N similar features (hereinafter referred to as the first similar feature) belongs to the first channel of audio data, and the first channel of audio data and the first channel of video data are extracted from the first channel of live audio and video data. In this case, the present application can determine the video frame that satisfies the aforementioned audio and video correspondence in the first channel of video data as the positioning video frame based on the audio position information of the first similar feature in the first channel of audio data, and the audio and video correspondence between the first channel of audio data and the first channel of video data. It can be understood that the positioning video frame and the first similar feature satisfy the audio and video correspondence between the first channel of live audio and video data.
[0088] The live video server can determine N positioning video frames and the video position information of each positioning video frame in the corresponding video data from the N channels of video data in the above manner. The N positioning video frames belong to the N channels of video data, and the N positioning video frames correspond to the same audio content. Thereafter, the live video server can modify the display timestamp corresponding to each channel of video data according to the server timestamp and the video position information corresponding to the N positioning video frames, so that the N channels of video data can be aligned in the time dimension.
[0089] For example, when N=3, the position information corresponding to the first similar feature is the first position, and the first similar feature belongs to the first audio data. Similarly, the second similar feature corresponds to the second position, and the second similar feature belongs to the second audio data. The third similar feature corresponds to the third position, and the third similar feature belongs to the third audio data. In this case, the PTS of the first video data at the first position, the PTS of the second video data at the second position, and the PTS of the third video data at the third position are the same, for example, they can all be the current timestamp of the live video server. After determining the PTS of the first video data at the first position, the live video server can use this as a reference to modify the PTS of the first video data at other positions. Similarly, the live video server can modify the PTS of the second video data at other positions based on the PTS of the second video data at the second position. The live video server can modify the PTS of the third video data at other positions based on the PTS of the third video data at the third position.
[0090] In some embodiments, after aligning the live audio and video data of each channel based on the audio alignment result, the method further includes:
[0091] Step C1: for each channel of the aligned N channels of live audio and video data, encode the live audio and video data according to a plurality of preset encoding specifications, and obtain a plurality of live encoded data; wherein the encoding specifications include resolution and frame rate;
[0092] Step C3: Push the live video to the live terminal according to various live encoding data.
[0093] In this embodiment, taking into account the diversity of live broadcast terminals and the different decoding capabilities of different live broadcast terminals, the live broadcast video server can provide multi-perspective videos with multiple encoding specifications to meet the multi-perspective video playback requirements of live broadcast terminals with different decoding capabilities, thereby providing low-latency multi-perspective videos to the live broadcast terminals and improving the playback smoothness of the multi-perspective videos.
[0094] In this embodiment, after aligning N channels of live audio and video data, the live video server can perform video encoding according to preset M encoding rules for the aligned N channels of live audio and video data, and obtain multiple live encoding data, where M is a positive integer greater than 1.
[0095] It can be understood that each channel of live audio and video data corresponds to M types of live encoding data, and the M types of live encoding data correspond one to one with the M encoding rules. After encoding and processing N channels of live audio and video data, N×M types of live encoding data can be obtained. For example, if N=2, M=3, and the first encoding rule is: resolution 720P / frame rate 30, the second encoding specification is resolution 1080P / frame rate 50, and the third encoding rule is: resolution 2160P / frame rate 60, then the first channel of live audio and video data can be encoded according to the first encoding rule, the second encoding rule and the third encoding rule, and three types of live encoding data can be obtained. Similarly, the second channel of live audio and video data can be encoded according to the first encoding rule, the second encoding rule and the third encoding rule, and three types of live encoding data can be obtained.
[0096] Furthermore, in one example, during the encoding process, the live video server can write the timestamp of the live video server into the PES header PTS corresponding to the audio and video frame, and the live terminal can parse the PTS to achieve synchronous playback of multi-view images. Wherein, PES refers to the packetized elementary stream obtained after the elementary audio and video streams are packaged and grouped according to the PS / TS specification, and a single PES includes a PES header and elementary stream data.
[0097] The live video server can push live video to the live terminal according to the capability of the live terminal, so that the live terminal can receive live audio and video data from N perspectives. It should be noted that for the N channels of live audio and video data received by the same live terminal, the encoding specifications corresponding to every two channels of live audio and video data can be the same or different, and this application does not impose specific restrictions on this.
[0098] In some embodiments, live video is pushed to a live terminal according to various live encoding data, including:
[0099] Based on the predetermined multicast address information, various live broadcast encoded data are multicasted to push live video to the live broadcast terminal.
[0100] Specifically, the existing technology usually pushes multi-view videos in the form of on-demand, and some solutions use video slicing transmission protocols such as HLS (HTTP Live Streaming, an adaptive bitrate streaming media transmission protocol based on HTTP) to complete it, resulting in a delay of more than 10 seconds in live broadcast. To solve this problem, this embodiment uses multicast to broadcast multi-view videos, thereby saving transmission bandwidth on the basis of meeting the diversity of live broadcast terminal decoding capabilities, thereby reducing live broadcast delay and achieving low-latency live broadcast.
[0101] The live video server can transmit video encoding data with different encoding rules for different viewing angles, video encoding data with the same encoding rules for different viewing angles, and video encoding data with different encoding rules for the same viewing angle to the network through a multicast protocol and different multicast address information to achieve live video push. The multicast address information may include an IP (Internet Protocol) address and a port. In one example, the multicast address may be numbered at fixed port intervals.
[0102] In some embodiments, according to various live encoding data, live video is pushed to the live terminal, further comprising:
[0103] Step D1: receiving a live broadcast request sent by a live broadcast terminal; wherein the live broadcast request carries decoding capability information of the live broadcast terminal;
[0104] Step D3: determining the target encoding specification according to the decoding capability information, and generating request response information based on the target multicast address information corresponding to the target encoding specification;
[0105] Step D5: Return request response information to the live broadcast terminal.
[0106] In this embodiment, considering that the live broadcast terminal is not clear about the association between the various videos when joining the multicast, it is difficult for the live broadcast terminal to determine the multicast it needs to join in order to simultaneously watch multiple videos from different perspectives. Based on this, the live broadcast video server can match the playback capabilities of the live broadcast terminal and provide it with a multicast playback function for multi-perspective videos, while also meeting the service requirements of a large number of users accessing, low-latency live broadcasting, and interactive real-time performance.
[0107] Specifically, the live broadcast terminal can report its own decoding capability information to the live broadcast video server through a live broadcast request. When the live broadcast video server receives the live broadcast request, it can extract the decoding capability information corresponding to the live broadcast terminal, and determine the target encoding specification that matches the capability of the live broadcast terminal based on the decoding capability information, and determine the target multicast address information accordingly, so that the live broadcast terminal can receive the multi-view video corresponding to the target encoding specification based on the target multicast address information.
[0108] Furthermore, after obtaining the multi-view video corresponding to the target encoding rule, the live broadcast terminal can decode the received multi-view video and play it according to the PTS of each video, thereby displaying the time-aligned multi-view video.
[0109] In this way, there is no need for the live video server to mix the audio and video collected from multiple channels. The live terminal can obtain the live video stream that matches its own playback capability through the live playback request, and can achieve synchronous playback of multi-view videos by parsing the PTS information.
[0110] In some embodiments, obtaining a multi-view video includes:
[0111] According to multiple preset paths, each channel of live audio and video data is read respectively; wherein the multiple preset paths correspond one to one to the multiple channels of audio and video data.
[0112] In this embodiment, different video capture devices can transmit the live audio and video data collected by the devices to the live video server in real time according to the fixed path negotiated with the live video server. In this way, the live video server can read out multiple live audio and video code streams collected in real time according to the negotiated fixed path to achieve the acquisition of original multi-view video.
[0113] To facilitate understanding of the solution of the present application, a specific example is provided below for illustration.
[0114] In one example, if Figure 3 , Figure 4 and Figure 5 As shown, the multi-view video processing method of this example may include the following steps:
[0115] S501: 4 video acquisition devices respectively acquire 4 channels of live audio and video data, and transmit the acquired live audio and video data to the live video server in real time according to the agreed path; further, RTP (Real-time Transport Protocol) can be used to transmit live audio and video data. In this way, in the video acquisition stage, the acquired audio and video content is transmitted to the video server by using the existing video equipment acquisition and transmission scheme without changing the acquisition device.
[0116] S502: The live video server receives and stores 4 channels of live audio and video stream data, and separates 4 channels of audio data from the 4 channels of live audio and video stream data respectively.
[0117] S503: The live video server parses the 4-channel audio data, identifies the same / similar audio features from the 4-channel audio data, and writes the server timestamp into the PTS of the PES header. In this step, after identifying the same / similar audio features, the live video server uses this as the starting time point, marks the PES header information at the aligned audio positions of each channel and the video frames synchronized with the audio, rewrites the server timestamp into the PTS, and then fills the PTS of each header with the server timestamp.
[0118] S504: The live video server encodes and transcodes according to different encoding rules to form video streams of different resolutions. In this step, three encoding specifications can be used for encoding, such as specification 1: resolution 720P / frame rate 30, specification 2: resolution 1080P / frame rate 50, and specification 3: resolution 2160P / frame rate 60.
[0119] S505: The live video server transmits multiple channels and multiple specifications of audio and video streams to the network. For example, the multicast IP address of the first channel of live video stream data is: 239.45.3.41, and the video streams of three encoding specifications are transmitted to ports 5100 / 5102 / 5104 respectively; the multicast IP address of the second video stream data is: 239.45.3.42, and the streams of three specifications are transmitted to ports 5100 / 5102 / 5104 respectively; and so on.
[0120] S506: The live broadcast terminal sends a live broadcast request with capability information. For example, the live broadcast request sent via HTTP (Hypertext Transfer Protocol) may pass parameters "name": "frame1", "capability": "2160P / 60fps", "name": "frame2", "capability": "720P / 30fps".
[0121] S507: The live video server matches the decoding capability of the terminal and returns the target multicast address information to the live terminal. For example, after receiving the above live broadcast request, the HTTP response returned by the live video server to the live terminal may pass parameters "view_num": 4, "view_angle": "0 / 90 / 180 / 270", "view_IP": "239.45.3.41 / 239.45.3.42 / 239.45.3.43 / 239.45.3.44", "view_Port": "frame1-5104 / frame2-5100".
[0122] S508: The live broadcast terminal selects a viewing angle, joins the corresponding multicast address, and parses the PTS to synchronously present multi-view images. For example, the live broadcast terminal can join the multicast according to the multicast addresses igmp: / / 239.45.3.41 / 5104 and igmp: / / 239.45.3.42 / 5100 to obtain a 4K video stream with a viewing angle of 0 degrees and a 720P video stream with a viewing angle of 90 degrees.
[0123] This application proposes a method and system for providing multicast playback for multi-view videos. This method performs feature recognition on audio collected from multiple perspectives, achieves audio alignment, adds PTS timestamp information to each video, and performs video encoding with multiple different resolutions, which are transmitted through a multicast protocol in the same or different IP and different ports. The terminal obtains the corresponding live video stream according to its own playback capability, and realizes multi-view video decoding and synchronous playback. This patent solves the problem of audio and video synchronization of multi-view videos through audio synchronization technology based on feature recognition, multicast transmission, terminal capability matching, etc., and meets the needs of various types of terminals with different capabilities to play multi-view videos. It is suitable for live broadcast of multi-view videos using multicast protocols, and provides solutions for multicast service scenarios in the field of visual networking.
[0124] The following describes a multi-view video processing device provided in an embodiment of the present application. The multi-view video processing device described below and the multi-view video processing method described above can refer to each other.
[0125] In some embodiments, Figure 6 As shown, the embodiment of the present application further provides a multi-view video processing device 600, comprising:
[0126] The video acquisition module 602 is used to acquire multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of the live audio and video data includes audio data and video data, N is a positive integer and N≥2;
[0127] The audio alignment result acquisition module 604 is used to process the multi-channel audio data using an audio alignment algorithm and obtain an audio alignment result;
[0128] A video alignment module 606, configured to align the multiple channels of video data based on the audio alignment result and obtain a video alignment result;
[0129] The timestamp modification module 608 is used to modify the display timestamps corresponding to the video data of each channel according to the video alignment result, so that the video frames at the same position correspond to the same display timestamps.
[0130] In some embodiments, the audio alignment algorithm includes an audio feature extraction algorithm and an audio feature matching algorithm. The audio alignment result acquisition module 604 of the present application includes:
[0131] A feature extraction unit, used to extract audio features corresponding to each channel of audio data using the audio feature extraction algorithm;
[0132] A feature matching unit is used to use the audio feature matching algorithm to match the audio features corresponding to each channel of the audio data, and obtain the audio alignment result based on the matching result; wherein, the audio alignment result includes N similar features and audio position information of each similar feature in the corresponding audio data, and the N similar features belong to N channels of audio data.
[0133] In some embodiments, the video alignment result includes N positioning video frames and video position information of each positioning video frame in the corresponding video data; the video alignment module 606 of the present application includes:
[0134] A positioning video frame determination unit is used to, for each of the similar features, take the live audio and video data to which the similar feature belongs as the target audio and video data, and the audio position information corresponding to the similar feature as the target position information; based on the audio and video correspondence of the target audio and video data, take the video frame corresponding to the target position information in the video data of the target audio and video data as a positioning video frame, and determine the video position information of the positioning video frame.
[0135] The timestamp modification module 608 of the present application includes:
[0136] The PTS modification unit is used to modify the display timestamp corresponding to each channel of the video data according to the server timestamp and the video position information corresponding to the N positioning video frames.
[0137] In some embodiments, the multi-view video processing device 600 of the present application further includes:
[0138] The encoding module is used to encode each channel of the aligned N channels of live audio and video data according to a plurality of preset encoding specifications, and obtain a plurality of live encoded data; wherein the encoding specifications include resolution and frame rate;
[0139] The push module is used to push the live video to the live terminal according to the various live encoding data.
[0140] In some embodiments, the push module of the present application includes:
[0141] The multicast unit is used to multicast the various live broadcast encoded data based on the predetermined multicast address information to push the live broadcast video to the live broadcast terminal.
[0142] In some embodiments, the push module of the present application further includes:
[0143] A request receiving unit, configured to receive a live broadcast request sent by the live broadcast terminal; wherein the live broadcast request carries the decoding capability information of the live broadcast terminal;
[0144] a target address determination unit, configured to determine a target coding specification according to the decoding capability information, and generate request response information based on target multicast address information corresponding to the target coding specification;
[0145] A response returning unit is used to return the request response information to the live broadcast terminal.
[0146] In some embodiments, the video acquisition module 602 of the present application includes:
[0147] The data reading unit is used to read the live audio and video data of each channel according to multiple preset paths; wherein the multiple preset paths correspond one-to-one to the multiple channels of audio and video data.
[0148] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-view video processing method in any embodiment.
[0149] In one embodiment, the present application also provides a live video server, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-view video processing method in any embodiment.
[0150] Indicatively, Figure 7The internal structure diagram of a live video server provided in an embodiment of the present application is shown in FIG. Figure 7 The live video server 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by a memory 901, for storing instructions executable by the processing component 902, such as an application. The application stored in the memory 901 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute instructions to perform the steps of the multi-view video processing method described in any of the above embodiments.
[0151] The live video server 900 may further include a power supply component 903 configured to perform power management of the live video server 900, a wired or wireless network interface 904 configured to connect the live video server 900 to the network, and an input / output (I / O) interface 905. The live video server 900 may operate based on an operating system stored in the memory 901, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0152] Those skilled in the art can understand that the internal structure of the live video server shown in the present application is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the live video server to which the scheme of the present application is applied. The specific live video server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.
[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0155] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing multi-view video, characterized in that: include: Acquire a multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of the live audio and video data includes audio data and video data, N is a positive integer and N≥2; Using an audio alignment algorithm to process the multiple channels of audio data and obtain an audio alignment result; Aligning the multiple channels of video data based on the audio alignment result, and obtaining a video alignment result; According to the video alignment result, the display timestamps corresponding to the video data of each channel are modified respectively, so that the video frames at the same position correspond to the same display timestamps.
2. The method according to claim 1, characterized in that The audio alignment algorithm includes an audio feature extraction algorithm and an audio feature matching algorithm; The method of using an audio alignment algorithm to process the multiple channels of audio data and obtaining an audio alignment result includes: Using the audio feature extraction algorithm, respectively extract the audio features corresponding to each channel of the audio data; Using the audio feature matching algorithm, the audio features corresponding to the audio data of each channel are matched, and the audio alignment result is obtained based on the matching result; The audio alignment result includes N similar features and audio position information of each similar feature in corresponding audio data, and the N similar features belong to N channels of audio data.
3. The method according to claim 2, characterized in that The video alignment result includes N positioning video frames and video position information of each positioning video frame in the corresponding video data; The aligning the multiple channels of video data based on the audio alignment result and obtaining the video alignment result includes: For each of the similar features, the live audio and video data to which the similar feature belongs is used as the target audio and video data, and the audio position information corresponding to the similar feature is used as the target position information. According to the audio and video correspondence relationship of the target audio and video data, the video frame corresponding to the target position information in the video data of the target audio and video data is used as a positioning video frame, and the video position information of the positioning video frame is determined; The step of respectively modifying the display timestamps corresponding to the video data of each channel according to the video alignment result includes: According to the server timestamp and the video position information corresponding to the N positioning video frames, the display timestamp corresponding to each channel of the video data is modified respectively.
4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: For each channel of the aligned N channels of live audio and video data, encode the live audio and video data according to a plurality of preset encoding specifications, and obtain a plurality of live encoded data; wherein the encoding specifications include resolution and frame rate; According to the various live broadcast encoding data, live broadcast video is pushed to the live broadcast terminal.
5. The method according to claim 4, characterized in that The method of pushing live video to a live terminal according to various live encoding data includes: Based on the predetermined multicast address information, the various live broadcast encoded data are multicast transmitted to push the live broadcast video to the live broadcast terminal.
6. The method according to claim 5, characterized in that The method of pushing live video to a live broadcast terminal according to the various live broadcast encoding data further includes: Receiving a live broadcast request sent by the live broadcast terminal; wherein the live broadcast request carries decoding capability information of the live broadcast terminal; Determine a target encoding specification according to the decoding capability information, and generate request response information based on target multicast address information corresponding to the target encoding specification; Return the request response information to the live broadcast terminal.
7. The method according to any one of claims 1 to 3, characterized in that: The obtaining of multi-view video includes: According to a plurality of preset paths, live audio and video data collected by a plurality of video collection devices are received respectively; wherein the plurality of preset paths correspond one-to-one to the plurality of channels of live audio and video data.
8. A multi-view video processing device, characterized in that: include: A video acquisition module, used to acquire multi-view video; wherein the multi-view video includes N channels of live audio and video data acquired at different viewpoints, each channel of the live audio and video data includes audio data and video data, N is a positive integer and N≥2; An audio alignment result acquisition module is used to process the multiple channels of audio data using an audio alignment algorithm and obtain an audio alignment result; A video alignment module, used for aligning the multiple channels of video data based on the audio alignment result, and obtaining a video alignment result; The timestamp modification module is used to modify the display timestamps corresponding to the video data of each channel according to the video alignment result, so that the video frames at the same position correspond to the same display timestamps.
9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the multi-view video processing method as described in any one of claims 1 to 7.
10. A live video server, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the multi-view video processing method as described in any one of claims 1 to 7 are performed.
Citation Information
Patent Citations
Multi-video synchronization method and device
CN112995708A
Video processing method and device, electronic equipment and computer readable storage medium
CN113542795A
Directing method, device and equipment and computer storage medium
CN114339302A
Multi-channel audio and video synchronous live broadcast method and system
CN115474083A
Methods and apparatus for device capabilities discovery and utilization within a content-based network
US20080201748A1
Cited By
Video playing method, computer equipment and readable storage medium
CN120769125A
