Multi-view Video Transmission Method, Apparatus, Processing Device, Display Device, and Medium

Abstract

The present disclosure provides a multi-view video transmission method, apparatus, processing device, display device, and medium, which relate to the field of network transmission technologies. Among them, the multi-view video transmission method includes: obtaining multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of the video images corresponding to a shooting angle; selecting multiple consecutive views in the multi-angle array based on the number of views of a sliding window; splicing the video images of the multiple consecutive views based on a downsampling operation to generate a multi-grid video image; performing compression encoding processing on the multi-grid video image so as to send the processed multi-grid video image to a playback device. Through the technical solution of the present disclosure, by reasonably setting the number of views in the sliding window, while improving the quality of the video image of a single downsampled view, it is ensured that all the video of multiple groups of all angles collected can be selected and transmitted.

Description

Background Art

[0002] In order to enhance the user's immersive experience, multi-perspective technology has come into being. This technology supports users to freely change the viewing perspective, breaking through the problem of limited viewing perspective and non-interaction of traditional video viewing, and has been applied in major sports and variety shows.

[0003] In related technologies, the implementation process of multi-view video transmission mainly includes: collecting video images of all views at the same frame time, stitching them into a large multi-view video image, and then compressing and encoding them for transmission. For example, if 4K multi-view video is stitched, 9 views are arranged, and the video image of a single view needs to be downsampled to 720P (4k / 9), or Figure 1 As shown, 16 viewing angles need to be downsampled to 540P (4k / 16).

[0004] The above scheme has the following defects: on the one hand, due to the limitation of bandwidth and decoding capability of the equipment, the stitching resolution has a certain upper limit requirement. The more viewing angles are arranged, the fewer pixels can be allocated to a single viewing angle in the stitched image, and the lower the resolution of a single viewing angle. On the other hand, if the video quality of a single viewing angle is to be guaranteed, the number of viewing angles, the viewing angle range or the viewing angle difference can only be limited, which in turn affects the display effect of the single viewing angle video image.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a multi-view video transmission method, device, processing equipment, display device and storage medium, which at least to a certain extent overcome the problem that the related art cannot take into account both the number of view angles and the quality of single-view video images.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a multi-view video transmission method is provided, which is applied to a processing device and includes: obtaining multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of the video images corresponding to a shooting angle; selecting a plurality of consecutive views in the multi-angle array based on the number of views of a sliding window, the sliding window being configured to slide in an array region corresponding to the multi-angle array, and the number of views of the sliding window being less than the total number of views of the multi-angle array; splicing the video images of the plurality of consecutive views based on a downsampling operation to generate a multi-grid video image; performing compression encoding processing on the multi-grid video image to send the processed multi-grid video image to a playback device.

[0009] In one embodiment, the number of views of the sliding window is less than or equal to a window threshold, and the window threshold is configured based on at least one of a playback resolution of a playback device, a transmission resolution of the processing device, and a splicing resolution of the multi-grid video image.

[0010] In one embodiment, the plurality of consecutive views include a reference view, and selecting a plurality of consecutive views in the multi-angle array based on the number of views of the sliding window specifically includes: selecting adjacent views of the reference view in the multi-angle array based on the number of views of the sliding window and a selection rule to obtain the plurality of consecutive views.

[0011] In one embodiment, before selecting adjacent views of the reference view in the multi-angle array based on the number of views of the sliding window and the selection rule, it further includes: receiving a view setting instruction sent by a playback device, and determining the corresponding reference view based on the view setting instruction; or determining one of the medians of the number of views of the sliding window as the serial number of the reference view.

[0012] In one embodiment, selecting adjacent views of the reference view in the multi-angle array based on the number of views of the sliding window and the selection rule specifically includes: when the selection rule is suitable for selecting views on both sides adjacent to the reference view, determining a first parameter and a second parameter based on the number of views of the sliding window; determining the adjacent views based on the relationship between the corresponding serial number of the reference view in the multi-angle and the first parameter and / or the second parameter.

[0013] In one embodiment, determining the first parameter and the second parameter based on the number of views of the sliding window specifically includes: calculating the first parameter based on a first calculation formula; calculating the second parameter based on a second calculation formula, where the first calculation formula is used to take the largest integer less than or equal to of, and the second calculation formula is used to take the smallest integer greater than or equal to The smallest integer, where S is the number of perspectives of the sliding window.

[0014] In one embodiment, determining the adjacent perspective based on the relationship between the corresponding serial number in the multi - perspectives based on the reference perspective and the first parameter and / or the second parameter specifically includes: when the difference between the corresponding serial number and the first parameter is less than 1, configuring the serial number of the adjacent perspective based on the number of perspectives of the sliding window; when the sum of the corresponding serial number and the second parameter is greater than the total number of perspectives of the multi - perspective array, configuring the serial number of the adjacent perspective within a first numerical range, where the first numerical range is greater than the difference between the total number of perspectives and the number of perspectives and less than or equal to the total number of perspectives; when the corresponding serial number is in other numerical ranges, configuring the serial number of the adjacent perspective within a second numerical range, where the second numerical range is greater than or equal to the difference between the corresponding serial number and the first parameter and less than or equal to the sum of the corresponding serial number and the second parameter.

[0015] In one embodiment, selecting the adjacent perspective of the reference perspective in the multi - perspective array based on the number of perspectives of the sliding window and the selection rule specifically includes: when the selection rule is suitable for selecting a side perspective adjacent to the reference perspective, determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives.

[0016] In one embodiment, determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives specifically includes: when the adjacent perspective is preset to the left of the reference perspective, detecting whether the number of left - hand perspectives of the reference perspective satisfies the number of perspectives of the sliding window; if it satisfies the number of perspectives of the sliding window, selecting multiple consecutive perspectives from the left - hand perspectives as the adjacent perspective; if it does not satisfy the number of perspectives of the sliding window, determining the remaining number of perspectives based on the number of perspectives of the sliding window and the number of left - hand perspectives, so as to select all the left - hand perspectives and the adjacent right - hand perspectives of the remaining number of perspectives.

[0017] In one embodiment, determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives specifically includes: when the adjacent perspective is preset to the right of the reference perspective, detecting whether the number of right - hand perspectives of the reference perspective satisfies the number of perspectives of the sliding window; if it satisfies the number of perspectives of the sliding window, selecting multiple consecutive perspectives from the right - hand perspectives as the adjacent perspective; if it does not satisfy the number of perspectives of the sliding window, determining the remaining number of perspectives based on the number of perspectives of the sliding window and the number of right - hand perspectives, so as to select all the right - hand perspectives and the adjacent left - hand perspectives of the remaining number of perspectives.

[0018] In one embodiment, it further includes: when receiving an adjustment instruction for changing the reference viewing angle, determining the adjusted reference viewing angle based on the adjustment instruction, and reconfiguring the video image viewing angle within the sliding window.

[0019] According to another aspect of the present disclosure, there is provided a multi-view video transmission method, which is applied to a display device and includes: receiving a multi-grid video image sent by a processing device; determining a playback viewing angle in the multi-grid video image based on a user's viewing angle selection operation, so as to perform playback based on the playback viewing angle.

[0020] In one embodiment, before receiving the multi-grid video image sent by the processing device, it further includes: receiving a user's viewing angle selection operation; generating a corresponding viewing angle setting instruction based on the viewing angle selection operation; sending the viewing angle setting instruction to the processing device, so that when receiving the multi-grid video image, directly determining the playback viewing angle based on the viewing angle setting instruction.

[0021] In one embodiment, the determining the playback viewing angle in the multi-grid video image based on the user's viewing angle selection operation specifically includes: displaying the received multi-grid video image; receiving the viewing angle selection operation on the multi-grid video image; determining the playback viewing angle based on the viewing angle selection operation.

[0022] In one embodiment, it further includes: generating a viewing angle adjustment area, where the viewing angle adjustment area includes a multi-angle array and a sliding window that slides within the multi-angle array; receiving a user's adjustment operation on the sliding window to generate a corresponding adjustment instruction based on the adjustment operation; sending the adjustment instruction to the processing device, so that the processing device adjusts the reference viewing angle based on the adjustment instruction.

[0023] In one embodiment, the performing playback based on the playback viewing angle specifically includes: extracting a to-be-played video image corresponding to the playback viewing angle in the multi-grid video image; performing an upsampling operation on the to-be-played video image based on a playback resolution to generate a playback video, and displaying the playback video.

[0024] According to another aspect of the present disclosure, there is provided a multi-view video transmission device, including: an acquisition module configured to acquire multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of the video images corresponding to a shooting angle; a selection module configured to select a plurality of consecutive views in the multi-angle array based on the number of views of a sliding window, the sliding window being configured to slide in an array area corresponding to the multi-angle array, the number of views of the sliding window being less than the total number of views of the multi-angle array; a splicing module configured to splice the video images of the plurality of consecutive views based on a downsampling operation to generate a multi-grid video image; and a sending module configured to perform compression encoding processing on the multi-grid video image to send the processed multi-grid video image to a playback device.

[0025] According to yet another aspect of the present disclosure, there is provided a multi-view video transmission device, including: a receiving module configured to receive a multi-grid video image sent by a processing device; a determination module configured to determine a playback view in the multi-grid video image based on a user's view selection operation; an extraction module configured to extract a to-be-played video image corresponding to the playback view in the multi-grid video image; and a display module configured to perform an upsampling operation on the to-be-played video image based on a playback resolution to generate a playback video and display the playback video.

[0026] According to yet another aspect of the present disclosure, there is provided a processing device, including: a processor; and a memory configured to store executable instructions of the processor; the processor being configured to execute the multi-view video transmission method of the first aspect above by executing the executable instructions.

[0027] According to yet another aspect of the present disclosure, there is provided a display device, including: a processor; and a memory configured to store executable instructions of the processor; the processor being configured to execute the multi-view video transmission method of the second aspect above by executing the executable instructions.

[0028] According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the multi-view video transmission method above.

[0029] The multi-view video transmission method and transmission device provided by the embodiments of the present disclosure set a sliding window with a certain number of viewing angles. When multiple groups of video images synchronously collected by a camera group arranged in a multi-angle array are acquired, based on the number of viewing angles set by the sliding window, multiple consecutive viewing angles are selected in the multi-angle array to splice the video images of these multiple consecutive viewing angles to obtain a multi-grid video image. After further compression coding processing, it is sent to a playback device, realizing the selection of at least a part of all the multiple groups of video images for splicing. The setting of the sliding window, on the one hand, by reasonably setting the number of viewing angles in the sliding window, can ensure the quality of the video image of a single downsampled viewing angle on the premise that the resolution of the multi-grid video image is limited. On the other hand, by sliding the sliding window in the array area corresponding to the multi-angle array, multi-angle videos with a specified number of viewing angles can be arbitrarily selected from multiple groups of video images, so that the number of angles arranged in the multi-angle array does not need to be set too small, thus being able to take into account both the number of shooting viewing angles and the viewing range.

[0030] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0032] Figure 1 Showing a flowchart of a multi-view video transmission method in the related art;

[0033] Figure 2 Showing a flowchart of a multi-view video transmission method in an embodiment of the present disclosure;

[0034] Figure 3 Showing a flowchart of another multi-view video transmission scheme in an embodiment of the present disclosure;

[0035] Figure 4 Showing a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0036] Figure 5 Showing a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0037] Figure 6 Showing a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0038] Figure 7Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0039] Figure 8 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0040] Figure 9 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0041] Figure 10 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0042] Figure 11 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0043] Figure 12 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0044] Figure 13 Shows a flowchart of yet another multi-view video transmission method in an embodiment of the present disclosure;

[0045] Figure 14 Shows a schematic diagram of a multi-view video transmission device in an embodiment of the present disclosure;

[0046] Figure 15 Shows a schematic diagram of another multi-view video transmission device in an embodiment of the present disclosure;

[0047] Figure 16 Shows a block diagram of the structure of a computer device in an embodiment of the present disclosure; and

[0048] Figure 17 Shows a block diagram of the structure of a program product in an embodiment of the present disclosure. Detailed implementation manners

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0050] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0051] The solution provided by the present application, by setting a sliding window with a certain number of viewing angles, when collecting multiple groups of video images synchronously collected by a camera group arranged in a multi-angle array, based on the number of viewing angles set by the sliding window, selects multiple consecutive viewing angles in the multi-angle array to splice the video images of these multiple consecutive viewing angles to obtain a multi-grid video image. After further compression coding processing, it is sent to a playback device, realizing the selection of at least a part of all the multiple groups of video images for splicing. The setting of the sliding window, on the one hand, by reasonably setting the number of viewing angles in the sliding window, can ensure the quality of the video image of a single downsampled viewing angle on the premise that the resolution of the multi-grid video image is limited. On the other hand, by sliding the sliding window in the array area corresponding to the multi-angle array, multiple-angle videos with a specified number of viewing angles can be arbitrarily selected from multiple groups of video images, so that the number of angles arranged in the multi-angle array does not need to be set too small, thus being able to take into account both the number of shooting viewing angles and the viewing range.

[0052] For ease of understanding, the terms (abbreviations) involved in the present application are first explained below.

[0053] Downsampling, that is, shrinking an image. For an image I with a size of M*N, performing s-fold downsampling on it means obtaining a lower-resolution image with a size of (M / s)*(N / s). Of course, s should be a common divisor of M and N. If considering an image in matrix form, it is to turn the image within an s*s window of the original image into one pixel.

[0054] Upsampling, that is, enlarging an image. Image enlargement almost always uses the interpolation method, that is, inserting new elements between pixel points using a suitable interpolation algorithm based on the original image pixels.

[0055] To solve the above problems, the present disclosure proposes a method for realizing super-frequency fusion based on flexible spectrum fusion.

[0056] Next, each step of the multi-view video transmission method in the present exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

[0057] Figure 2The flowchart of a multi-view video transmission method in an embodiment of the present disclosure is shown.

[0058] As Figure 2 shown, the multi-view video transmission method according to an embodiment of the present disclosure is applied to a processing device and includes the following steps:

[0059] Step S202, obtaining multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of video images corresponding to a camera angle.

[0060] Among them, by obtaining multiple groups of video images collected by the camera group, synchronous acquisition of the video images captured by the camera group is achieved.

[0061] Step S204, based on the number of perspectives of the sliding window, selecting multiple consecutive perspectives in the multi-angle array. The sliding window is configured to slide in the array area corresponding to the multi-angle array, and the number of perspectives of the sliding window is less than the total number of perspectives of the multi-angle array.

[0062] Among them, the reference perspective can be a perspective determined based on a received perspective setting instruction, or the perspective at the middle position in the sliding window can be used as the reference perspective.

[0063] In addition, the sliding window is configured to slide in the array area corresponding to the multi-angle array. That is to say, the multiple perspectives within the sliding perspective are multiple consecutive perspectives in the multi-angle array.

[0064] Those skilled in the art can also understand that the number of perspectives of the sliding window can be flexibly set.

[0065] In one embodiment, the multiple consecutive perspectives include the reference perspective. Based on the number of perspectives of the sliding window, selecting multiple consecutive perspectives in the multi-angle array specifically includes: selecting the adjacent perspectives of the reference perspective in the multi-angle array based on the number of perspectives of the sliding window and the selection rule to obtain multiple consecutive perspectives.

[0066] Among them, the position of the reference perspective can be at the leftmost end, the rightmost end or the middle position of the sliding window.

[0067] In this embodiment, by setting the reference perspective, the reference perspective can correspond to the perspective selected by the user on the video display device, or the perspective initially set on the processing device. Determine multiple consecutive perspectives including the reference perspective within the sliding window to achieve the selection of consecutive perspectives.

[0068] Specifically, in one embodiment, before selecting adjacent views of the reference view in the multi-angle array based on the number of views and selection rules of the sliding window, it further includes: receiving a view setting instruction sent by the playback device, and determining the corresponding reference view based on the view setting instruction; or determining one of the medians of the number of views of the sliding window as the serial number of the reference view.

[0069] Among them, the view setting instruction sent by the playback device can be directly generated by the video image view selected by the user.

[0070] As Figure 3 shown, the camera array is determined based on the information of the camera group arranged in the multi-angle array, and the video image of the target scene or object is collected. Correspondingly, a sliding buffer window is configured as the sliding window. The number of views of the sliding window is S, and the number of angles of the camera array is N. S is less than N. Preferably, S is less than or equal to 9. The views within the sliding window are determined by sliding the sliding window reciprocally in the display area corresponding to the camera array.

[0071] Step S206: Stitch the video images of multiple consecutive views based on the downsampling operation to generate a multi-grid video image.

[0072] Among them, by performing downsampling processing on the video image, the downsampling operation is performed on the collected video of the original pixels to reduce the original video image to 1 / S of the original size, where S is the number of views of the sliding window, and further through stitching, a multi-grid video image is obtained.

[0073] Step S208: Perform compression encoding processing on the multi-grid video image to send the processed multi-grid video image to the playback device.

[0074] Among them, by performing compression encoding processing on the multi-grid video image, the reliable transmission of the multi-grid video image is ensured.

[0075] In the present disclosure, the compression encoding is specifically reversible compression encoding processing.

[0076] In this embodiment, by setting a sliding window with a certain number of viewing angles, when multiple groups of video images synchronously collected by a camera group arranged in a multi-angle array are acquired, based on the number of viewing angles set in the sliding window, multiple consecutive viewing angles are selected from the multi-angle array to splice the video images of these multiple consecutive viewing angles to obtain a multi-grid video image. After further compression coding processing, it is sent to a playback device, realizing the selection of at least a part of all the multiple groups of video images for splicing. The setting of the sliding window, on the one hand, by reasonably setting the number of viewing angles in the sliding window, can improve the quality of the video image of a single downsampled viewing angle on the premise that the resolution of the multi-grid video image is limited. On the other hand, by sliding the sliding window in the array area corresponding to the multi-angle array, multi-angle videos with a specified number of viewing angles can be arbitrarily selected from multiple groups of video images, and the video images of all angles can be selected and spliced and transmitted, so that the number of angles arranged in the multi-angle array does not need to be set too small, thus being able to balance the number of viewing angles and the viewing range of shooting. On the further hand, the amount of data of the multiple groups of video images originally collected is large. If all are synchronously transmitted to the display device, the processing amount is large, and the calculation and transmission costs are high. In the present disclosure, a sliding window smaller than the original number is used for image splicing, which can be deployed at a 5G edge computing node, facilitating the improvement of the real-time performance of transmission.

[0077] Specifically, as a preferred implementation manner, the reference viewing angle is determined based on the viewing angle selected by the user. The splicing resolution of the multi-grid video image is 4K. An adaptive sliding window is constructed for the video data to be transmitted. According to the user's current viewing angle, data of multiple consecutive viewing angles adjacent to the current viewing angle are adaptively selected and stored in the sliding window, and then the data of multiple viewing angles in the window are spliced to generate a 4K multi-grid video. In this way, it is not necessary to splice the video images of all the viewing angles collected. When downsampling, it can be ensured that the pixels of the single-view video image in the spliced video image are still in a good display quality, so that the saved pixels can improve the video quality of the viewing angles within the window range.

[0078] Furthermore, since the reference viewing angle can be determined based on the viewing angle setting instruction sent by the playback device, that is, the sliding window can respond in real time to the viewing angle setting instruction generated based on the user operation sent by the display device, it will not affect the global number of viewing angles of the user. And based on the adjustment instruction, any viewing angle in the global viewing angle can be used as the reference viewing angle for selecting the sliding window. Therefore, on the premise that the playback resolution or the transmission resolution is determined, the problem of fewer collected viewing angles due to the limited 4K resolution in the related art is effectively improved.

[0079] In one embodiment, the number of viewing angles of the sliding window is less than or equal to the window threshold, and the window threshold is configured based on at least one of the playback resolution of the playback device, the transmission resolution configuration of the processing device, and the splicing resolution of the multi-grid video image.

[0080] Among them, the playback resolution of the playback device can be sent to the processing device in advance, or determined by the processing device based on the playback resolution defined by the industry.

[0081] As Figure 4 shown, in one embodiment, in step S204, a specific implementation manner of selecting adjacent perspectives of the reference perspective in the multi-angle array based on the number of perspectives and selection rules of the sliding window includes:

[0082] Step S402, when the selection rule is suitable for selecting the perspectives on both sides of the adjacent reference perspective, determine the first parameter and the second parameter based on the number of perspectives of the sliding window.

[0083] Among them, the selection rule is suitable for selecting the perspectives on both sides of the adjacent reference perspective, specifically, with the reference perspective as the center, the adjacent perspectives are not on both sides of the reference perspective.

[0084] Step S404, determine the adjacent perspective based on the relationship between the corresponding serial number of the reference perspective in the multi-angles and the first parameter and / or the second parameter.

[0085] Among them, let S be the number of perspectives of the sliding window, d be the serial number of the reference perspective, N be the total number of all perspectives collected, △l be the first parameter, and △r be the second parameter.

[0086] In this embodiment, by determining the first parameter and the second parameter based on the number of perspectives of the sliding window, after determining the serial number of the reference perspective, the other adjacent perspectives within the sliding window can be determined based on the relationship between the serial number of the reference perspective and the first parameter and / or the second parameter, so as to ensure the reasonable layout of different perspectives within the sliding window.

[0087] As Figure 5 shown, in one embodiment, in step S402, determining the first parameter and the second parameter based on the number of perspectives of the sliding window specifically includes:

[0088] Step S502, calculate the first parameter based on the first calculation formula.

[0089] Step S504, calculate the second parameter based on the second calculation formula.

[0090] Among them, the first calculation formula is used to take the largest integer less than or equal to , and the second calculation formula is used to take the smallest integer greater than or equal to , and S is the number of perspectives of the sliding window.

[0091] Specifically, the first calculation formula is that is, take the largest integer within the number indicated by the less than or equal to symbol, and the second calculation formula is That is, take the smallest integer greater than or equal to the number within the greater than or equal to symbol.

[0092] In one embodiment, based on the relationship between the corresponding serial number in multiple perspectives with respect to the reference perspective and the first parameter and / or the second parameter, determine the adjacent perspective, specifically including:

[0093] Step S506, when the difference between the corresponding serial number and the first parameter is less than 1, configure the serial number of the adjacent perspective based on the number of perspectives in the sliding window.

[0094] Step S508, when the sum of the corresponding serial number and the second parameter is greater than the total number of perspectives in the multi-angle array, configure the serial number of the adjacent perspective within the first numerical range, where the first numerical range is greater than the difference between the total number of perspectives and the number of perspectives, and less than or equal to the total number of perspectives.

[0095] Step S510, when the corresponding serial number is in other numerical ranges, configure the serial number of the adjacent perspective within the second numerical range, where the second numerical range is greater than or equal to the difference between the corresponding serial number and the first parameter, and less than or equal to the sum of the corresponding serial number and the second parameter.

[0096] Specifically, determine the serial number of the perspective within the sliding window based on the following formula.

[0097]

[0098] Wherein, D i is the perspective identifier within the sliding window, and i is the serial number of the corresponding perspective.

[0099] In this embodiment, after determining the reference perspective, based on the relationship between the serial number of the reference perspective, the first parameter, and the second parameter, determine other perspectives adjacent to the reference perspective. On the one hand, it can ensure that the video images of multiple consecutive perspectives are within the sliding window, thus facilitating the user to select the desired viewing perspective. On the other hand, by fully considering the serial number range where the serial number of the reference perspective is located, it can ensure the reliability of the perspective selection of the sliding window, especially preventing abnormal selection when the reference perspective is at both ends.

[0100] In one embodiment, in step S204, another specific implementation manner of selecting the adjacent perspective of the reference perspective in the multi-angle array based on the number of perspectives in the sliding window and the selection rule includes:

[0101] When the selection rule is suitable for selecting the perspective on one side adjacent to the reference perspective, determine the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives.

[0102] As Figure 6 shown, in one embodiment, determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives specifically includes:

[0103] Step S602, when the adjacent view is preset to the left relative to the reference view, detect whether the number of left views of the reference view meets the number of views of the sliding window.

[0104] Step S604, if the number of views of the sliding window is met, select multiple consecutive views from the left views as the adjacent views.

[0105] Step S606, if the number of views of the sliding window is not met, determine the remaining number of views based on the number of views of the sliding window and the number of left views, so as to select all the left views and the adjacent right views of the remaining number of views.

[0106] As Figure 7 shown, in one embodiment, determining the adjacent views based on the preset position and the number of views of the adjacent view relative to the reference view specifically includes:

[0107] Step S702, when the adjacent view is preset to the right relative to the reference view, detect whether the number of right views of the reference view meets the number of views of the sliding window.

[0108] Step S704, if the number of views of the sliding window is met, select multiple consecutive views from the right views as the adjacent views.

[0109] Step S706, if the number of views of the sliding window is not met, determine the remaining number of views based on the number of views of the sliding window and the number of right views, so as to select all the right views and the adjacent left views of the remaining number of views.

[0110] In this embodiment, the reference view can also be used as the reference view at the end of the sliding window. That is to say, the serial number of the reference view is placed at the leftmost or rightmost end of the sliding window. Assuming it is placed at the leftmost end, if the number of adjacent views on the right meets the remaining number of views, it is determined that the reference view is placed at the leftmost end. If not, the insufficient part is filled in the adjacent right side, and the reference view is shifted to the right. Assuming it is placed at the rightmost end, the setting method is opposite to that at the leftmost end. This method can also obtain multiple consecutive and reliable views.

[0111] In one embodiment, it further includes: when receiving an adjustment instruction to change the reference view, determining the adjusted reference view based on the adjustment instruction and reconfiguring the video image views within the sliding window.

[0112] In this embodiment, through the interaction with the display device, the instructions sent by the display device can be parsed. When the instruction is an adjustment instruction, the corresponding playback perspective is determined based on the adjustment instruction as the reference perspective. Based on the above-mentioned selection method of adjacent perspectives, the perspective that is most adjacent to the adjusted reference perspective is selected into the sliding window, where both the number of perspectives and the perspective range of the sliding window can be dynamically adjusted.

[0113] Further, by splicing the video images corresponding to the perspectives within the sliding window, a multi-grid video image is generated. On the one hand, it is not necessary to splice the video images of all perspectives. On the other hand, when the user needs to view the video images of other perspectives, the user can send an adjustment instruction through the above-mentioned display device.

[0114] After the processing device receives the adjustment instruction, the reference perspective is adjusted by adjusting the parameters of the reference perspective. After the reference perspective is adjusted, the adjacent perspectives of the sliding window are also adjusted accordingly.

[0115] Furthermore, on the display device side, the perspective switching range of a single user operation can be limited within the current sliding window, but the sliding window is dynamically updated, which does not affect the user's global perspective range.

[0116] As Figure 8 shown, a multi-perspective video transmission method according to another embodiment of the present disclosure is applied to a display device and includes:

[0117] Step S802: Receive the multi-grid video image sent by the processing device.

[0118] Step S804: Determine the playback perspective in the multi-grid video image based on the user's perspective selection operation.

[0119] Among them, the user's perspective selection operation on the multi-grid video image can be implemented before receiving the multi-grid video image or after receiving the multi-grid video image. When it is implemented before receiving the multi-grid video image, a generated perspective setting instruction is correspondingly sent to the processing device.

[0120] Step S806: Extract the to-be-played video image corresponding to the playback perspective in the multi-grid video image.

[0121] Step S808: Perform an upsampling operation on the to-be-played video image based on the playback resolution to generate a playback video and display the playback video.

[0122] In this embodiment, by setting a sliding window with a certain number of viewing angles at the processing device end, when multiple groups of video images synchronously collected by a camera group arranged in a multi-angle array are acquired, based on the number of viewing angles set by the sliding window, multiple consecutive viewing angles are selected from the multi-angle array to splice the video images of these multiple consecutive viewing angles to obtain a multi-grid video image. The display device receives the multi-grid video image, and through the user's viewing angle selection operation, the playback viewing angle can be determined based on the user's selection for video playback. This multi-grid video image can ensure the resolution of the spliced image while also ensuring the display quality of the video image of a single viewing angle in the spliced video image.

[0123] According to the multi-view video transmission solution of the present disclosure, on the one hand, according to the user operation, adaptively select the data of multiple consecutive viewing angles that are closest to the current viewing angle and store them in the sliding window, and then splice the data of multiple viewing angles in the window into a multi-grid 4K video, without the need to splice the video images of all viewing angles; on the other hand, the saved pixels can improve the resolution of the viewing angles within the window range. Under the limited transmission resolution, the video quality of the viewing angles can be improved; on the further hand, the sliding window responds to the user operation in real time, does not affect the user's global viewing angle range, does not limit the total number of viewing angles, and is conducive to expanding the viewing angle range and reducing the viewing angle difference.

[0124] As Figure 9 shown, the multi-view video transmission method according to another embodiment of the present disclosure, which is applied to a display device, includes:

[0125] Step S902, receive the user's viewing angle selection operation.

[0126] Step S904, generate a corresponding viewing angle setting instruction based on the viewing angle selection operation.

[0127] Step S906, send the viewing angle setting instruction to the processing device.

[0128] Step S908, when receiving the multi-grid video image, directly determine the playback viewing angle based on the viewing angle setting instruction.

[0129] In this embodiment, before receiving the multi-grid video image, the display device can receive the user's viewing angle selection operation, generate a corresponding viewing angle setting instruction, and send the viewing angle setting instruction to the processing device. The processing device can then set the reference viewing angle according to the viewing angle setting instruction, further determine the adjacent viewing angles based on the reference viewing angle, and splice them into a multi-grid video image. When receiving the multi-grid video image, it can automatically determine the playback viewing angle based on the previously generated viewing angle setting instruction to play the video based on the playback viewing angle, which can simplify the user's operation steps.

[0130] As Figure 10As shown, a multi-view video transmission method according to another embodiment of the present disclosure is applied to a display device and includes:

[0131] Step S1002, receiving a multi-grid video image sent by a processing device.

[0132] Among them, the generation of the multi-grid video image can be based on a setting instruction of a reference view sent in advance, or can be based on a reference view automatically selected by the processing device.

[0133] Step S1004, displaying the received multi-grid video image.

[0134] Step S1006, receiving a view selection operation on the multi-grid video image.

[0135] Step S1008, determining a playback view based on the view selection operation.

[0136] In this embodiment, after receiving the multi-grid video image, a view selection operation of the user on the multi-grid video image is received, and this method makes the selection of the playback view more reliable.

[0137] As Figure 11 shown, in one embodiment, it further includes:

[0138] Step S1102, generating a view adjustment area, where the view adjustment area includes a multi-angle array and a sliding window sliding in the multi-angle array.

[0139] Step S1104, receiving an adjustment operation of the user on the sliding window to generate a corresponding adjustment instruction based on the adjustment operation.

[0140] Step S1106, sending the adjustment instruction to the processing device so that the processing device adjusts the reference view based on the adjustment instruction.

[0141] In this embodiment, by generating an adjustment operation on the sliding window, when the user needs to view video images of other views, they can do so by sending an adjustment instruction. In this way, at the processing device side, it is not necessary to splice video images of all views, and only the multi-grid video image generated by the selected partial views that the user is willing to view needs to be selected, and switching can be performed at any time.

[0142] In addition, those skilled in the art can understand that the multi-view video transmission solution of the present disclosure is applied to a system in which a camera, a processing device, and a display device interact, and this system can realize real-time live broadcast of multi-angle video images.

[0143] Specifically, in the application of free-viewpoint technology, on the display device side, a viewpoint adjustment area is generated to support users to freely drag and change the viewpoint, enabling multi-angle viewing of video content, breaking through the problems of limited viewing angles and lack of operability in traditional video viewing, and can be used in scenarios such as sports live broadcasts and variety show live broadcasts. Furthermore, it can improve the video quality of free-viewpoint technology under the condition of limited bandwidth and device decoding capabilities; and on the premise of not restricting the number of viewpoints collected, it is conducive to expanding the viewpoint range of free-viewpoint technology and reducing the viewpoint difference.

[0144] As Figure 12 shown, according to the multi-angle video transmission system of the present disclosure, it includes a collection module 1202, a splicing module 1204, and an interaction and playback module 1206. Among them, the collection module is specifically a camera group, the splicing module is specifically a processing device, and the interaction and playback module is specifically a display device.

[0145] Among them, the collection module 1202 is used to collect multi-angle video images of the camera array and synchronize them;

[0146] The splicing module 1204 is specifically used for:

[0147] (1) Parse the instruction sent by the interaction and playback module 1206 to determine the current reference viewpoint and use it as the playback viewpoint.

[0148] (2) Determine other adjacent viewpoints within the sliding window according to the reference viewpoint, and the number and range of viewpoints in the window can be dynamically adjusted.

[0149] (3) Splice the viewpoints within the sliding window into a multi-grid video image; as Figure 3 shown, for the current viewpoint D7 and a window size of 9, no matter how many viewpoints are arranged, always select the 9 viewpoints closest to the current viewpoint into the sliding window and splice them.

[0150] The interaction and playback module 1206 is specifically used for:

[0151] (1) Receive the spliced multi-viewpoint video, send a viewpoint adjustment instruction to the splicing module, and intercept and play the video image of the playback viewpoint.

[0152] (2) Obtain the operation information of the user's adjusted viewpoint. After detecting a viewpoint change, send an adjustment instruction with the adjusted viewpoint parameters to the splicing module.

[0153] (3) Limit the viewpoint switching range of the user's single operation within the current sliding window. Since the sliding window is dynamically updated, it does not affect the user's global viewpoint range.

[0154] As Figure 13 shown, in combination with Figure 12, According to the multi-view video transmission solution of the present disclosure, it specifically includes:

[0155] Step S1302, the splicing module sets a sliding window.

[0156] Among them, let the window size, that is, the number of views S within the sliding window, and the total number of views be N (numbered 1, 2,..., N).

[0157] Step S1304, the acquisition module acquires multiple groups of video images from N views of the camera array and synchronously transmits them to the splicing module.

[0158] Step S1306, the splicing module receives multiple groups of video images from multiple views synchronously acquired by the acquisition module and determines multiple consecutive views within the current sliding window.

[0159] Specifically, let d be the serial number of the reference view, and select the S - 1 views closest to d and place them within the sliding window.

[0160] Among them, initially, the sliding window takes views [1, S], d can take the median of S, or the initial view range of the sliding window can be determined according to the median view. If the view is updated, according to the current view parameter d, the views Di within the sliding window are determined according to the following formula. The values of Δl and Δr are related to the window median, and the two can be interchanged.

[0161] In addition, the serial number d of the reference view can also be determined by the interaction and playback module.

[0162]

[0163] Among them, D i is the view identifier within the sliding window, and i is the serial number of the corresponding view.

[0164] Step S1308, the splicing module splices the video images of multiple views within the sliding window through downsampling operations to generate a multi-grid video image.

[0165] Specifically, downsample the video images of each view within the sliding window to a specified resolution (this value is determined by S and the transmission resolution. For example, for a 4K and 9 - view sliding window, it corresponds to 720P), splice the downsampled video images to form a large multi-grid video image, perform compression encoding, and transmit it to the interaction and playback module for display.

[0166] Step S1310, the interaction and playback module receives the multi-view video stream and view range parameters from the splicing module and determines the current playback view.

[0167] Step S1312: The interaction and playback module intercepts the video image of the current playback perspective from the multi-grid video image, and transforms it to the resolution set by the device through upsampling for playback.

[0168] Step S1314: The interaction and playback module monitors the user operations. If the user drags to switch the current playback perspective, an adjustment instruction is generated and sent to the splicing module.

[0169] Specifically, the user can drag left or right to change the perspective, and the perspective range is limited within the current sliding window.

[0170] Step S1316: The splicing module monitors the instructions sent by the interaction and playback module. When receiving the adjustment instruction, it returns to Step S1306 to re-determine multiple consecutive perspectives within the current sliding window.

[0171] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0172] Next, refer to Figure 14 to describe the multi-perspective video transmission device 1400 according to the embodiments of the present invention. Figure 14 The multi-perspective video transmission device 1400 shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0173] The multi-perspective video transmission device 1400 is presented in the form of hardware modules. The components of the multi-perspective video transmission device 1400 may include, but are not limited to: an acquisition module 1402 for acquiring multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of video images corresponding to a camera angle; a selection module 1404 for selecting multiple consecutive perspectives in the multi-angle array based on the number of perspectives of the sliding window, the sliding window being configured to slide in the array area corresponding to the multi-angle array, and the number of perspectives of the sliding window being less than the total number of perspectives of the multi-angle array; a splicing module 1406 for splicing the video images of multiple consecutive perspectives based on downsampling operations to generate a multi-grid video image; a sending module 1408 for performing compression encoding processing on the multi-grid video image to send the processed multi-grid video image to the playback device.

[0174] In one embodiment, the number of perspectives of the sliding window is less than or equal to a window threshold, and the window threshold is configured based on at least one of the playback resolution of the playback device, the transmission resolution configuration of the processing device, and the splicing resolution of the multi-grid video image.

[0175] In one embodiment, the multiple consecutive viewpoints include a reference viewpoint, and the selection module 1404 is further configured to: based on the number of viewpoints of the sliding window and the selection rule, select the adjacent viewpoints of the reference viewpoint in the multi-viewpoint array to obtain multiple consecutive viewpoints.

[0176] In one embodiment, it further includes: a receiving module 1410, configured to receive a viewpoint setting instruction sent by a playback device, and determine a corresponding reference viewpoint based on the viewpoint setting instruction; or a determining module 1412, configured to determine one of the medians of the number of viewpoints of the sliding window as the serial number of the reference viewpoint.

[0177] In one embodiment, the selection module 1404 is further configured to: when the selection rule is suitable for selecting the viewpoints on both sides of the adjacent reference viewpoint, determine a first parameter and a second parameter based on the number of viewpoints of the sliding window; determine the adjacent viewpoints based on the relationship between the corresponding serial number of the reference viewpoint in the multi-viewpoints and the first parameter and / or the second parameter.

[0178] In one embodiment, the selection module 1404 is further configured to: calculate the first parameter based on a first calculation formula; calculate the second parameter based on a second calculation formula, where the first calculation formula is used to take the largest integer less than or equal to and the second calculation formula is used to take the smallest integer greater than or equal to and S is the number of viewpoints of the sliding window.

[0179] In one embodiment, the selection module 1404 is further configured to: when the difference between the corresponding serial number and the first parameter is less than 1, configure the serial number of the adjacent viewpoint based on the number of viewpoints of the sliding window; when the sum of the corresponding serial number and the second parameter is greater than the total number of viewpoints of the multi-viewpoint array, configure the serial number of the adjacent viewpoint within a first numerical range, where the first numerical range is greater than the difference between the total number of viewpoints and the number of viewpoints and less than or equal to the total number of viewpoints; when the corresponding serial number is in other numerical ranges, configure the serial number of the adjacent viewpoint within a second numerical range, where the second numerical range is greater than or equal to the difference between the corresponding serial number and the first parameter and less than or equal to the sum of the corresponding serial number and the second parameter.

[0180] In one embodiment, the selection module 1404 is further configured to: when the selection rule is suitable for selecting the viewpoint on one side of the adjacent reference viewpoint, determine the adjacent viewpoint based on the preset position of the adjacent viewpoint relative to the reference viewpoint and the number of viewpoints.

[0181] In one embodiment, the selection module 1404 is further configured to: when the adjacent view is preset to the left relative to the reference view, detect whether the number of left views of the reference view meets the number of views of the sliding window; if the number of left views meets the number of views of the sliding window, select multiple consecutive views from the left views as the adjacent views; if the number of left views does not meet the number of views of the sliding window, determine the remaining number of views based on the number of views of the sliding window and the number of left views, so as to select all the left views and the adjacent right views with the remaining number of views.

[0182] In one embodiment, the selection module 1404 is further configured to: when the adjacent view is preset to the right relative to the reference view, detect whether the number of right views of the reference view meets the number of views of the sliding window; if the number of right views meets the number of views of the sliding window, select multiple consecutive views from the right views as the adjacent views; if the number of right views does not meet the number of views of the sliding window, determine the remaining number of views based on the number of views of the sliding window and the number of right views, so as to select all the right views and the adjacent left views with the remaining number of views.

[0183] In one embodiment, it further includes: an adjustment module 1414, configured to, when receiving an adjustment instruction for changing the reference view, determine the adjusted reference view based on the adjustment instruction, and reconfigure the video image views within the sliding window.

[0184] The following refers to Figure 15 to describe the multi-view video transmission device 1500 according to the embodiments of the present invention. Figure 15 The multi-view video transmission device 1500 shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0185] The multi-view video transmission device 1500 is presented in the form of a hardware module. The components of the multi-view video transmission device 1500 may include but are not limited to: a receiving module 1502, configured to receive the multi-grid video image sent by the processing device; a determining module 1504, configured to determine the playing view in the multi-grid video image based on the user's view selection operation, so as to play based on the playing view.

[0186] In one embodiment, before receiving the multi-grid video image sent by the processing device, it further includes: the receiving module 1502 is further configured to: receive the user's view selection operation; a first generating module 1506, configured to generate a corresponding view setting instruction based on the view selection operation; a sending module 1508, configured to send the view setting instruction to the processing device, so as to directly determine the playing view based on the view setting instruction when receiving the multi-grid video image.

[0187] In one embodiment, the determination module 1504 is further configured to: display the received multi-grid video image; receive a perspective selection operation on the multi-grid video image; and determine a playback perspective based on the perspective selection operation.

[0188] In one embodiment, it further includes: a second generation module 1510, configured to generate a perspective adjustment area, where the perspective adjustment area includes a multi-angle array and a sliding window that slides in the multi-angle array; the receiving module 1506 is further configured to: receive an adjustment operation of the user on the sliding window to generate a corresponding adjustment instruction based on the adjustment operation; the sending module 1508 is further configured to: send the adjustment instruction to a processing device, so that the processing device adjusts a reference perspective based on the adjustment instruction.

[0189] In one embodiment, it further includes: an extraction module 1512, configured to extract a to-be-played video image corresponding to the playback perspective from the multi-grid video image; a playback module 1514, configured to perform an upsampling operation on the to-be-played video image based on a playback resolution to generate a playback video and display the playback video.

[0190] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation manner, a complete software implementation manner (including firmware, microcode, etc.), or an implementation manner combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0191] The following refers to Figure 16 to describe the computer device 1600 according to this embodiment of the present invention. Figure 16 The computer device 1600 shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0192] As Figure 16 shown, the computer device 1600 is presented in the form of a general-purpose computing device. The components of the computer device 1600 may include, but are not limited to: at least one of the above-mentioned processing units 1610, at least one of the above-mentioned storage units 1620, and a bus 1630 connecting different system components (including the storage unit 1620 and the processing unit 1610).

[0193] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1610, so that the processing unit 1610 executes the steps according to various exemplary embodiments of the present invention described in the above "exemplary method" part of this specification. For example, the processing unit 1610 can execute the solution described in steps S202 to S208 as Figure 2 shown.

[0194] The storage unit 1620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 16201 and / or a cache storage unit 16202, and may further include a read-only storage unit (ROM) 16203.

[0195] The storage unit 1620 may also include a program / utilities 16204 having a set (at least one) of program modules 16205. Such program modules 16205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0196] The bus 1630 may represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0197] The computer device 1600 may also communicate with one or more external devices 1670 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 1600, and / or may communicate with any device that enables the computer device 1600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 1650. Moreover, the computer device 1600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1660. As shown in the figure, the network adapter 1660 communicates with other modules of the computer device 1600 through the bus 1630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0198] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0199] In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of this specification. In some possible implementation manners, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0200] Referring Figure 17 As shown, a program product 1700 for implementing the above method according to an embodiment of the present invention is described. It may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0201] The program product may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0202] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0203] The program code contained on the readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0204] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0205] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more of the above-described modules or units can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0206] In addition, although the steps of the methods in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in that specific order, or that all of the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution, etc.

[0207] From the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the methods according to the embodiments of the present disclosure.

[0208] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the appended claims.

Claims

1. A multi-view video transmission method, characterized in that, applied to a processing device, including: obtaining multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of the video images corresponding to a shooting angle; selecting a plurality of consecutive views in the multi-angle array based on the number of views of a sliding window, the sliding window being configured to slide in an array area corresponding to the multi-angle array, the number of views of the sliding window being less than the total number of views of the multi-angle array, including: the plurality of consecutive views include a reference view, and based on the number of views of the sliding window and a selection rule, adjacent views of the reference view are selected in the multi-angle array to obtain the plurality of consecutive views, including: when the selection rule is suitable for selecting the two side views adjacent to the reference view, determining a first parameter and a second parameter based on the number of views of the sliding window; determining the adjacent views based on the relationship between the corresponding serial number of the reference view in the multi-angle and the first parameter and / or the second parameter; or when the selection rule is suitable for selecting one side view adjacent to the reference view, determining the adjacent view based on the preset position of the adjacent view relative to the reference view and the number of views; stitching the video images of the plurality of consecutive views based on a downsampling operation to generate a multi-grid video image; performing compression encoding processing on the multi-grid video image to send the processed multi-grid video image to a playback device.

2. The multi-view video transmission method according to claim 1, characterized in that, the number of views of the sliding window is less than or equal to a window threshold, and the window threshold is configured based on at least one of the playback resolution of the playback device, the transmission resolution of the processing device, and the stitching resolution of the multi-grid video image.

3. The multi-view video transmission method according to claim 1, characterized in that, before selecting the adjacent views of the reference view in the multi-angle array based on the number of views of the sliding window and the selection rule, further including: receiving a view setting instruction sent by the playback device, and determining the corresponding reference view based on the view setting instruction; or determining one of the medians of the number of views of the sliding window as the serial number of the reference view.

4. The multi-view video transmission method according to claim 1, characterized in that, the determining the first parameter and the second parameter based on the number of views of the sliding window specifically includes: calculating the first parameter based on a first calculation formula; calculating the second parameter based on a second calculation formula, wherein, the first calculation formula is used to obtain the largest integer less than or equal to , and the second calculation formula is used to obtain the smallest integer greater than or equal to , and S is the number of perspectives of the sliding window.

5. The multi-view video transmission method according to claim 4, characterized in that, the determining the adjacent views based on the relationship between the corresponding serial number of the reference view in the multi-angle and the first parameter and / or the second parameter specifically includes: when the difference between the corresponding serial number and the first parameter is less than 1, configuring the serial number of the adjacent view based on the number of views of the sliding window; When the sum of the corresponding serial number and the second parameter is greater than the total number of perspectives of the multi-angle array, configure the serial numbers of the adjacent perspectives within a first numerical range, where the first numerical range is greater than the difference between the total number of perspectives and the number of perspectives, and less than or equal to the total number of perspectives; When the corresponding serial number is within other numerical ranges, configure the serial numbers of the adjacent perspectives within a second numerical range, where the second numerical range is greater than or equal to the difference between the corresponding serial number and the first parameter, and less than or equal to the sum of the corresponding serial number and the second parameter.

6. The multi-perspective video transmission method according to claim 1, characterized in that determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives specifically includes: When the adjacent perspective is preset to the left of the reference perspective, detect whether the number of left perspectives of the reference perspective satisfies the number of perspectives of the sliding window; If the number of perspectives of the sliding window is satisfied, select multiple consecutive perspectives from the left perspectives as the adjacent perspective; If the number of perspectives of the sliding window is not satisfied, determine the remaining number of perspectives based on the number of perspectives of the sliding window and the number of left perspectives, so as to select all the left perspectives and the adjacent right perspectives of the remaining number of perspectives.

7. The multi-perspective video transmission method according to claim 1, characterized in that determining the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives specifically includes: When the adjacent perspective is preset to the right of the reference perspective, detect whether the number of right perspectives of the reference perspective satisfies the number of perspectives of the sliding window; If the number of perspectives of the sliding window is satisfied, select multiple consecutive perspectives from the right perspectives as the adjacent perspective; If the number of perspectives of the sliding window is not satisfied, determine the remaining number of perspectives based on the number of perspectives of the sliding window and the number of right perspectives, so as to select all the right perspectives and the adjacent left perspectives of the remaining number of perspectives.

8. The multi-perspective video transmission method according to any one of claims 3 to 7, characterized in that further comprising: When receiving an adjustment instruction to change the reference perspective, determine the adjusted reference perspective based on the adjustment instruction, and reconfigure the video image perspectives within the sliding window.

9. A multi-perspective video transmission method, characterized in that applied to a display device, comprising: Receives the multi-grid video image sent by the processing device. The processing device acquires multiple groups of video images collected by a camera group arranged in a multi-angle array, selects multiple consecutive perspectives in the multi-angle array based on the number of perspectives of the sliding window, splices the video images of the multiple consecutive perspectives based on downsampling operation to generate the multi-grid video image. Each group of the video images corresponds to a camera angle. The sliding window is configured to slide in the array area corresponding to the multi-angle array. The number of perspectives of the sliding window is less than the total number of perspectives of the multi-angle array. The multiple consecutive perspectives include a reference perspective. When selecting the two perspectives adjacent to the reference perspective on both sides, determine a first parameter and a second parameter based on the number of perspectives of the sliding window; determine the adjacent perspectives based on the relationship between the corresponding serial number of the reference perspective in the multi-angle and the first parameter and / or the second parameter. When selecting a perspective adjacent to the reference perspective on one side, determine the adjacent perspective based on the preset position of the adjacent perspective relative to the reference perspective and the number of perspectives; Determine the playback perspective in the multi-grid video image based on the user's perspective selection operation, and play based on the playback perspective.

10. The multi-perspective video transmission method according to claim 9, characterized in that, before receiving the multi-grid video image sent by the processing device, further includes: receiving the user's perspective selection operation; generating a corresponding perspective setting instruction based on the perspective selection operation; sending the perspective setting instruction to the processing device, so as to directly determine the playback perspective based on the perspective setting instruction when receiving the multi-grid video image.

11. The multi-perspective video transmission method according to claim 9, characterized in that, the determining the playback perspective in the multi-grid video image based on the user's perspective selection operation specifically includes: displaying the received multi-grid video image; receiving the perspective selection operation on the multi-grid video image; determining the playback perspective based on the perspective selection operation.

12. The multi-perspective video transmission method according to claim 9, characterized in that, further includes: generating a perspective adjustment area, which includes a multi-angle array and a sliding window sliding in the multi-angle array; receiving the adjustment operation of the user on the sliding window to generate a corresponding adjustment instruction based on the adjustment operation; sending the adjustment instruction to the processing device, so that the processing device adjusts the reference perspective based on the adjustment instruction.

13. The multi-perspective video transmission method according to any one of claims 9 to 12, characterized in that, the playing based on the playback perspective specifically includes: extracting the to-be-played video image corresponding to the playback perspective in the multi-grid video image; performing an upsampling operation on the to-be-played video image based on the playback resolution to generate a playback video, and displaying the playback video.

14. A multi-perspective video transmission device, characterized in that, applied to a processing device, includes: An acquisition module, configured to acquire multiple groups of video images collected by a camera group arranged in a multi-angle array, each group of the video images corresponding to a shooting angle; A selection module, configured to select multiple consecutive perspectives in the multi-angle array based on the number of perspectives of a sliding window, the sliding window being configured to slide in an array area corresponding to the multi-angle array, the number of perspectives of the sliding window being less than the total number of perspectives of the multi-angle array, including: the multiple consecutive perspectives include a reference perspective, and based on the number of perspectives of the sliding window and a selection rule, adjacent perspectives of the reference perspective are selected in the multi-angle array to obtain the multiple consecutive perspectives, including: when the selection rule is suitable for selecting perspectives on both sides adjacent to the reference perspective, a first parameter and a second parameter are determined based on the number of perspectives of the sliding window; based on the relationship between the corresponding serial number of the reference perspective in the multi-angles and the first parameter and / or the second parameter, the adjacent perspectives are determined; or when the selection rule is suitable for selecting a perspective on one side adjacent to the reference perspective, the adjacent perspective is determined based on a preset position of the adjacent perspective relative to the reference perspective and the number of perspectives; A splicing module, configured to splice the video images of the multiple consecutive perspectives based on a downsampling operation to generate a multi-grid video image; A sending module, configured to perform compression encoding processing on the multi-grid video image to send the processed multi-grid video image to a playback device.

15. A multi-perspective video transmission device characterized in that applied to a display device, comprising: A receiving module, configured to receive the multi-grid video image sent by a processing device, wherein the processing device acquires multiple groups of video images collected by a camera group arranged in a multi-angle array, selects multiple consecutive perspectives in the multi-angle array based on the number of perspectives of a sliding window, splices the video images of the multiple consecutive perspectives based on a downsampling operation to generate the multi-grid video image, each group of the video images corresponding to a shooting angle, the sliding window being configured to slide in an array area corresponding to the multi-angle array, the number of perspectives of the sliding window being less than the total number of perspectives of the multi-angle array, the multiple consecutive perspectives including a reference perspective, when selecting perspectives on both sides adjacent to the reference perspective, a first parameter and a second parameter are determined based on the number of perspectives of the sliding window; based on the relationship between the corresponding serial number of the reference perspective in the multi-angles and the first parameter and / or the second parameter, the adjacent perspectives are determined, when selecting a perspective on one side adjacent to the reference perspective, the adjacent perspective is determined based on a preset position of the adjacent perspective relative to the reference perspective and the number of perspectives; A determination module, configured to determine a playback perspective in the multi-grid video image based on a user's perspective selection operation, so as to perform playback based on the playback perspective.

16. A processing device characterized in that comprising: A processor; and a memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the multi-view video transmission method according to any one of claims 1 to 8 by executing the executable instructions.

17. A display device, characterized in that it includes: a processor; and a memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the multi-view video transmission method according to any one of claims 9 to 13 by executing the executable instructions.

18. A computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the multi-view video transmission method according to any one of claims 1 to 13 is implemented.

Images