Multi-view video coding method and device, electronic equipment and medium

Abstract

The invention provides a multi-viewpoint video coding method and device, electronic equipment and a medium, the multi-viewpoint video comprises a main viewpoint video and an auxiliary viewpoint video, the method comprises the following steps: obtaining the installation positions of a main viewpoint camera and an auxiliary viewpoint camera, establishing a main viewpoint coordinate system and an auxiliary viewpoint coordinate system, determining a spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system, acquiring a main viewpoint video frame sequence and an auxiliary viewpoint video frame sequence, and converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, and coding the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence by adopting a preset coding rule and the reference viewpoint video frame sequence to obtain coding information of the multi-viewpoint video. The main viewpoint video frame sequence is converted according to the spatial corresponding relation, the problem that common content between viewpoints is difficult to use can be solved, and the coding efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device, electronic device and medium for encoding multi-viewpoint videos. Background Art

[0002] With the development of technologies such as virtual reality (VR), augmented reality (AR) and panoramic video, multi-view content is growing exponentially. Specifically, the video of the same scene can be captured from multiple viewpoints, such as in a studio or with a camera array. The captured multi-view video is transmitted over the network and presented in a variety of ways at the terminal, increasing the audience's immersion and interactivity.

[0003] However, the exponential increase in the amount of multi-channel video data has brought great challenges to transmission. The existing multi-view video coding standards cannot effectively utilize the spatial relationship between different viewpoints. When the camera positions are too different, it is difficult to accurately obtain the similar parts between video frames captured from different viewpoints. The common content between viewpoints is difficult to utilize, which seriously affects the efficiency of multi-view video coding. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a method, device, electronic device and medium for encoding multi-viewpoint videos.

[0005] In a first aspect of the implementation of the present application, a method for encoding a multi-view video is first provided, wherein the multi-view video includes a main view video captured by a main view camera and an auxiliary view video captured by an auxiliary view camera, and the method includes:

[0006] Obtaining the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0007] Establishing a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and establishing an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera;

[0008] Determining a spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0009] Acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video;

[0010] Converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence;

[0011] The main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence are encoded using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

[0012] Optionally, the main viewpoint camera and the auxiliary viewpoint camera are arranged in a preset scene, the spatial correspondence relationship includes a rotation correspondence relationship and / or a translation correspondence relationship, and the spatial correspondence relationship between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, including:

[0013] Selecting at least one common feature point in the preset scene;

[0014] Determining the main viewpoint feature coordinates of the common feature point in the main viewpoint coordinate system according to the spatial relationship between the installation position of the main viewpoint camera and the common feature point;

[0015] Determining the auxiliary viewpoint feature coordinates of the common feature point in the auxiliary viewpoint coordinate system according to the spatial relationship between the installation position of the auxiliary viewpoint camera and the common feature point;

[0016] Taking the main viewpoint feature coordinates and the auxiliary viewpoint feature coordinates as a matching point pair of the common feature points;

[0017] The rotation correspondence and / or translation correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the matching point pair.

[0018] Optionally, converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence includes:

[0019] Acquire at least one main viewpoint video frame from the main viewpoint video frame sequence in the main viewpoint coordinate system;

[0020] Converting the main viewpoint video frame into a main viewpoint rotated video frame according to the rotation correspondence;

[0021] Converting the main viewpoint rotated video frame into a reference viewpoint video frame in the auxiliary viewpoint coordinate system according to the translation correspondence;

[0022] At least one of the reference viewpoint video frames is combined in time sequence to form the reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system.

[0023] Optionally, the preset coding rule includes an inter-frame coding rule and an inter-view coding rule, and the adopting the preset coding rule and the reference viewpoint video frame sequence to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence to obtain the coding information of the multi-viewpoint video includes:

[0024] Encoding the main viewpoint video frame sequence using the inter-frame coding rule to obtain main viewpoint video coding information;

[0025] Encoding the auxiliary viewpoint video frame sequence by using the inter-frame coding rule, the inter-viewpoint coding rule, and the reference viewpoint video frame sequence to obtain auxiliary viewpoint video coding information;

[0026] The encoding information of the multi-view video is determined according to the main view video encoding information and the auxiliary view video encoding information.

[0027] Optionally, the adopting the inter-frame coding rule, the inter-view coding rule, and the reference viewpoint video frame sequence to encode the auxiliary viewpoint video frame sequence to obtain the auxiliary viewpoint video coding information includes:

[0028] Encoding the auxiliary viewpoint video frame sequence using the inter-frame coding rule to obtain auxiliary viewpoint first coding information;

[0029] Encoding the auxiliary viewpoint video frame sequence using the inter-viewpoint coding rule and the reference viewpoint video frame sequence to obtain auxiliary viewpoint second coding information;

[0030] The auxiliary viewpoint video encoding information is determined according to the auxiliary viewpoint first encoding information and the auxiliary viewpoint second encoding information.

[0031] Optionally, the main viewpoint video and the auxiliary viewpoint video are collected within a preset time period, and after converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, the method further includes:

[0032] Acquire any one of the reference viewpoint video frames in the reference viewpoint video frame sequence;

[0033] Determine the area to be detected in the reference viewpoint video frame according to the spatial correspondence;

[0034] A sliding detection window is set in the area to be detected; the size of the sliding detection window is determined by the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; the sliding detection window includes a number of pixels to be detected in the reference viewpoint video frame;

[0035] In the process of the sliding detection window sliding in the area to be detected, obtaining the gray value of the pixel to be detected in the sliding detection window and the median of the gray values ​​of several pixels to be detected in the sliding detection window;

[0036] Determine a grayscale difference between a grayscale value of any pixel to be detected in the sliding detection window and a median of the grayscale values;

[0037] If the grayscale difference is greater than a preset threshold, the pixel to be detected is replaced with a pixel corresponding to the median of the grayscale value;

[0038] After the sliding detection window traverses the area to be detected, the reference viewpoint video frames formed after replacing the pixels to be detected are formed into a new reference viewpoint video frame sequence.

[0039] Optionally, in the process of the sliding detection window sliding in the area to be detected, obtaining the grayscale value of the pixel to be detected in the sliding detection window and the median of the grayscale values ​​of several pixels to be detected in the sliding detection window includes:

[0040] In the process of the sliding detection window sliding in the area to be detected, grayscale values ​​of a number of pixels to be detected in the sliding detection window are acquired in real time;

[0041] Sorting the grayscale values ​​of the pixels to be detected in ascending order;

[0042] The median of the sorted grayscale values ​​is used as the median of the grayscale values ​​of the pixels to be detected.

[0043] In a second aspect of the implementation of the present application, there is also provided a multi-view video encoding device, characterized in that the multi-view video includes a main view video captured by a main view camera and an auxiliary view video captured by an auxiliary view camera, and the device includes:

[0044] An installation position acquisition module, used to acquire the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0045] A coordinate system establishment module, used to establish a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and to establish an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera;

[0046] A spatial correspondence determination module, used to determine the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0047] A video frame sequence acquisition module, used to acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video;

[0048] A spatial conversion module, configured to convert the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence;

[0049] The encoding module is used to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence by using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

[0050] In the third aspect of the implementation of the present application, there is also provided an electronic device, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, it implements the multi-viewpoint video encoding method as described above.

[0051] In the fourth aspect of the implementation of the present application, a computer-readable storage medium is also provided, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the multi-viewpoint video encoding method as described above is implemented.

[0052] The embodiments of the present application have the following advantages:

[0053] In the embodiment of the present application, by obtaining the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, a main viewpoint coordinate system is established according to the installation position of the main viewpoint camera, and an auxiliary viewpoint coordinate system is established according to the installation position of the auxiliary viewpoint camera, and the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera. A main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video are obtained, and the main viewpoint video frame sequence in the main viewpoint coordinate system is converted into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, and the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence are encoded using a preset encoding rule and a reference viewpoint video frame sequence to obtain encoding information of a multi-viewpoint video. The present application first converts the main viewpoint video frame sequence into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system, and then uses a preset coding rule and the reference viewpoint video frame sequence to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence, which can solve the problem of difficulty in utilizing common content between viewpoints and improve coding efficiency, ensuring that the coding information of multi-viewpoint videos can be generated efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0055] Figure 1 This is a schematic diagram of a multi-viewpoint independent encoding and decoding process provided by an embodiment of the present application;

[0056] Figure 2 It is a schematic diagram of a framework of a multi-view video coding architecture provided by an embodiment of the present application;

[0057] Figure 3 This is a flowchart of a multi-view video encoding method provided by an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of a spatial relationship conversion process provided by an embodiment of the present application;

[0059] Figure 5 It is a schematic diagram of a framework of multi-viewpoint coding based on spatial relationship transformation provided by an embodiment of the present application;

[0060] Figure 6 is a schematic diagram of different viewpoint camera shooting angles provided by an embodiment of the present application;

[0061] Figure 7 is a schematic diagram of video frame detail information loss provided by an embodiment of the present application;

[0062] Figure 8 It is a schematic diagram of a framework of a multi-view video encoding method combining spatial relationship conversion and detail filling provided by an embodiment of the present application;

[0063] Fig. 9 It is a structural diagram of a multi-view video encoding device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced to each other without contradiction.

[0065] With the development of technologies such as virtual reality (VR), augmented reality (AR), and panoramic video, multi-viewpoint content is growing exponentially. Specifically, videos of the same scene can be captured from multiple viewpoints, such as in a studio or with a camera array. The captured multi-viewpoint video is transmitted over the network and presented in a variety of ways at the terminal, increasing the audience's immersion and interactivity. However, the exponential increase in the amount of multi-channel video data has brought great challenges to transmission. At present, a large amount of multi-viewpoint video content does not use dedicated coding standards, but is encoded using common standards (such as H.264 / AVC, H.265 / HEVC) for each viewpoint video.

[0066] It should be noted that the multi-view coding technology may be a technology that utilizes redundant information between different cameras or viewpoints to achieve efficient compression and coding. In the embodiment of the present application, the multi-view coding technology may be used to achieve coding of multi-view videos.

[0067] HEVC (High Efficiency Video Coding) is a new generation of video coding standard, which mainly includes transformation, quantization, entropy coding, intra-frame and inter-frame prediction modules. Compared with previous international standards, HEVC introduces new coding technology in each module to improve compression efficiency. In the embodiment of the present application, intra-frame coding or inter-frame coding can be performed according to the principle of HEVC.

[0068] MV-HEVC (MultiView-HEVC, multi-view high-efficiency video coding): is an advanced syntax extension of HEVC, which can be regarded as the implementation of multi-view coding on the high-efficiency video coding standard. In the embodiment of the present application, encoding can also be performed with reference to the principle of MV-HEVC.

[0069] Reference Figure 1 , shows a schematic diagram of a multi-viewpoint independent encoding and decoding process provided by an embodiment of the present application. For each viewpoint video from viewpoint 1, viewpoint 2 to viewpoint n, enter the general encoder and general decoder respectively, and then the decoding results of each viewpoint video are combined to perform multi-viewpoint video playback.

[0070] Since multi-view videos are aimed at the same scene, video frames from different viewpoints at the same time have many commonalities. Therefore, in order to utilize the correlation between multiple viewpoints, MV-HEVC (MultiView-HEVC) can be used for multi-view efficient video coding. The MV-HEVC standard introduces the concept of view, usually View 0 represents the main view, View 1 and others represent auxiliary views. The reference rule for main view coding is basic HEVC, and each frame of the auxiliary view is encoded on top of the basic HEVC, with an additional reference frame between viewpoints.

[0071] Reference Figure 2 , showing a schematic diagram of a framework of a multi-view video coding architecture provided by an embodiment of the present application. Figure 2 A three-view MV-HEVC coding architecture is shown, where view 0 is the main view. The main view is coded using HEVC, which can take into account the correlation between frames on the time axis, while the auxiliary view considers both the correlation between viewpoints and time. For example, at time T2 of viewpoint 2, the B frame considers both the P frame at time T0 of viewpoint 2 and the B frame at time T4 of viewpoint 2, as well as the B frame at time T2 of viewpoint 0, making use of the relationship between viewpoints.

[0072] But there are some problems in MV-HEVC:

[0073] First, MV-HEVC does not exploit the spatial relationship between different viewpoints. Figure 2 For example, when performing inter-frame coding on viewpoint 2, the correlation between frames in the time axis and between viewpoints is used. For example, when encoding the B frame at time T2 of viewpoint 2, both the P frame at time T0 of viewpoint 2 and the B frame at time T4 of viewpoint 2 are considered in terms of time, and the B frame at time T2 of viewpoint 0 is also considered, and the relationship between viewpoints is utilized. However, when utilizing the correlation between viewpoints, no spatial conversion is performed. For example, for the B frame at time T2 of viewpoint 2, the B frame at time T2 of viewpoint 0 is used during inter-frame coding, and the coding is performed by calculating the similarity between the B frame at time T2 of viewpoint 0 and the B frame at time T2 of viewpoint 2.

[0074] Although multiple viewpoints are aimed at the same scene, due to the different positions and angles of the cameras, the content in the video frames shot from multiple viewpoints often has fixed rotation and translation. When the rotation and translation are large, it is difficult to obtain similar parts between frames only through motion estimation and compensation technology, which will greatly limit the encoding effect.

[0075] Secondly, the video frames converted based on spatial relationships in MV-HEVC may lose details. Since the cameras at different viewpoints are placed at different positions and angles, the content captured by each viewpoint is different. Therefore, when converting a video frame from a certain viewpoint to another viewpoint, there may be a loss of details, which will cause a large difference in pixels between the two frames in the part where the details are lost, affecting the encoding efficiency between viewpoints.

[0076] Therefore, the embodiment of the present application proposes a method for encoding multi-viewpoint video. When encoding multi-viewpoint video, the spatial relationship between multiple viewpoints can be used to calculate the rotation and translation matrices according to the spatial position relationship of the cameras, and then the video frame of the main viewpoint is converted to the auxiliary viewpoint based on the two matrices. When the auxiliary viewpoint is inter-frame encoded, the similarity between the auxiliary viewpoint and the converted main viewpoint video frame is used to avoid the problem of difficulty in obtaining similar parts caused by the large difference in camera positions, thereby improving the efficiency of multi-viewpoint video encoding.

[0077] In addition, in an embodiment of the present application, in order to address the situation in which detail information is lost in the converted video frames, resulting in low coding efficiency between viewpoints of the part where details are lost, a sliding window and a discrimination threshold are set to fill in the lost detail information, thereby improving the coding efficiency between viewpoints. On the other hand, based on the impact of different rotations and translations on the position of the part where details are lost, an adaptive search area is performed to improve the efficiency of detail filling.

[0078] Reference Figure 3 , showing a step flow chart of a multi-view video encoding method provided by an embodiment of the present application, wherein the multi-view video includes a main view video captured by a main view camera and an auxiliary view video captured by an auxiliary view camera.

[0079] It should be noted that multi-view video (MVV) refers to a video of the same event or scene shot from multiple different perspectives. Multi-view video can provide a richer visual experience, allowing users to watch the same event or scene from different angles, enhancing immersion and interactivity. In the embodiment of the present application, it is necessary to encode the multi-view video.

[0080] The main viewpoint camera refers to the camera used to shoot the main viewpoint in a multi-viewpoint video system. The main viewpoint camera is usually located in the center of the scene and provides the most important viewpoint.

[0081] The main viewpoint video refers to the video shot by the main viewpoint camera. The main viewpoint video usually contains the main perspective of the scene and provides the most important visual information.

[0082] In an embodiment of the present application, the main viewpoint video can be captured by a main viewpoint camera.

[0083] Auxiliary viewpoint cameras refer to cameras used to shoot auxiliary viewpoints in a multi-viewpoint video system. Auxiliary viewpoint cameras are usually located around the main viewpoint camera to provide different viewpoints.

[0084] Auxiliary viewpoint video refers to the video shot by the auxiliary viewpoint camera. Auxiliary viewpoint video usually contains auxiliary perspectives of the scene, providing richer visual information.

[0085] In an embodiment of the present application, auxiliary viewpoint video can be captured by an auxiliary viewpoint camera.

[0086] The method specifically comprises the following steps:

[0087] S101: Obtaining the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0088] In an embodiment of the present application, the installation position of the main view camera and the installation position of the auxiliary view camera can be obtained. The installation position can be in the form of three-dimensional coordinates. The installation position can also include the orientation information of the main view camera and the auxiliary view camera. The auxiliary view camera can be one or more.

[0089] S102: establishing a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and establishing an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera.

[0090] In the embodiment of the present application, the main viewpoint coordinate system can be established according to the installation position of the main viewpoint camera, and the auxiliary viewpoint coordinate system can be established according to the installation position of the auxiliary viewpoint camera. In other words, the main viewpoint coordinate system based on the main viewpoint camera and the auxiliary viewpoint coordinate system based on the auxiliary viewpoint camera can be constructed according to the three-dimensional coordinates and orientation information of the main viewpoint camera and the auxiliary viewpoint coordinate system.

[0091] S103: Determine a spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera.

[0092] In an embodiment of the present application, the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system can be determined according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera.

[0093] In the specific implementation, the internal parameters (such as focal length, principal point position, etc.) and external parameters (such as rotation matrix and translation matrix) of each camera can be obtained through camera calibration. According to the spatial position of the camera, the rotation matrix and translation matrix between the main viewpoint camera and each auxiliary viewpoint camera are calculated. The rotation matrix and translation matrix are used as the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system.

[0094] S104: Acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video.

[0095] In an embodiment of the present application, a main viewpoint video frame sequence in a main viewpoint video and an auxiliary viewpoint video frame sequence in an auxiliary viewpoint video may be obtained, wherein the main viewpoint video and the auxiliary viewpoint video may be shot for the same scene at the same time.

[0096] In a specific implementation, the main viewpoint video can be converted into a main viewpoint video frame sequence, and the auxiliary viewpoint video can be converted into an auxiliary viewpoint video frame sequence, wherein the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence are main viewpoint video frames and auxiliary viewpoint video frames corresponding to the same time sequence.

[0097] S105: Converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence.

[0098] In the embodiment of the present application, the main viewpoint video frame sequence in the main viewpoint coordinate system may be converted into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence.

[0099] In a specific implementation, the reference viewpoint video frame sequence can be understood as a video frame sequence consisting of the main viewpoint video frames in the main viewpoint coordinate system and the projected video frames in the auxiliary viewpoint coordinate system corresponding to the same time sequence.

[0100] S106: Encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

[0101] In the embodiment of the present application, the preset coding rule and the reference viewpoint video frame sequence can be used to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence to obtain the coding information of the multi-viewpoint video. The preset coding rule can be a multi-viewpoint coding rule based on spatial relationship conversion.

[0102] In a specific implementation, when performing inter-frame coding on the auxiliary viewpoint, the similarity between the auxiliary viewpoint video frame and the reference viewpoint video frame corresponding to the same moment can be used for coding to avoid the problem of difficulty in obtaining similar parts caused by large differences in camera positions.

[0103] In the embodiment of the present application, by obtaining the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, a main viewpoint coordinate system is established according to the installation position of the main viewpoint camera, and an auxiliary viewpoint coordinate system is established according to the installation position of the auxiliary viewpoint camera, and the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera. A main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video are obtained, and the main viewpoint video frame sequence in the main viewpoint coordinate system is converted into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, and the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence are encoded using a preset encoding rule and a reference viewpoint video frame sequence to obtain encoding information of a multi-viewpoint video. The present application first converts the main viewpoint video frame sequence into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system, and then uses a preset coding rule and the reference viewpoint video frame sequence to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence, which can solve the problem of difficulty in utilizing common content between viewpoints and improve coding efficiency, ensuring that the coding information of multi-viewpoint videos can be generated efficiently and accurately.

[0104] In an optional embodiment of the present application, the main viewpoint camera and the auxiliary viewpoint camera are set in a preset scene, the spatial correspondence includes a rotation correspondence and / or a translation correspondence, and step S103 further includes the following sub-steps:

[0105] S11: selecting at least one common feature point in the preset scene;

[0106] S12: determining the main viewpoint feature coordinates of the common feature point in the main viewpoint coordinate system according to the spatial relationship between the installation position of the main viewpoint camera and the common feature point;

[0107] S13: determining the auxiliary viewpoint feature coordinates of the common feature point in the auxiliary viewpoint coordinate system according to the spatial relationship between the installation position of the auxiliary viewpoint camera and the common feature point;

[0108] S14: taking the main viewpoint feature coordinates and the auxiliary viewpoint feature coordinates as a matching point pair of the common feature points;

[0109] S15: Determine the rotation correspondence and / or translation correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the matching point pair.

[0110] In an embodiment of the present application, at least one common feature point can be selected in a preset scene, and then the main viewpoint feature coordinates of the common feature point in the main viewpoint coordinate system are determined based on the spatial relationship between the installation position of the main viewpoint camera and the common feature point, and then the auxiliary viewpoint feature coordinates of the common feature point in the auxiliary viewpoint coordinate system are determined based on the spatial relationship between the installation position of the auxiliary viewpoint camera and the common feature point.

[0111] Then, the main viewpoint feature coordinates and the auxiliary viewpoint feature coordinates can be used as a matching point pair of common feature points, and the rotation correspondence and / or translation correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system can be determined according to the matching point pair.

[0112] In the specific implementation, n viewpoints can be represented by View 0, View 1, ..., View n, and the corresponding camera positions are: P0, P1, ..., P n .

[0113] Assume View 0 as the main viewpoint and the other viewpoints as auxiliary viewpoints. According to the spatial position of the camera, obtain the rotation matrix R between the main viewpoint camera and each auxiliary viewpoint camera through camera calibration and other methods. 0,i and T 0,i Translation matrix, R 0,i ,T 0,i =Camera_Calibration(P0,P i ), where i represents View i, i=1,2,...,n.

[0114] This application obtains the rotation matrix and translation matrix between the main viewpoint camera and each auxiliary viewpoint camera through camera calibration and other methods, ensuring that the video frames can be accurately converted and encoded, thereby improving the efficiency and quality of multi-viewpoint video encoding; by selecting common feature points and determining their feature coordinates in different coordinate systems, the accuracy and reliability of matching point pairs are ensured, thereby improving the stability and precision of the system; by determining the rotation correspondence and translation correspondence, it is ensured that the video frames can be accurately converted and encoded, reducing the transmission of redundant information, and improving encoding efficiency and bandwidth utilization.

[0115] In an optional embodiment of the present application, step S105 further includes the following sub-steps:

[0116] S21: Acquire at least one main viewpoint video frame from the main viewpoint video frame sequence in the main viewpoint coordinate system;

[0117] S22: converting the main viewpoint video frame into a main viewpoint rotated video frame according to the rotation correspondence;

[0118] S23: converting the main viewpoint rotated video frame into a reference viewpoint video frame in the auxiliary viewpoint coordinate system according to the translation correspondence relationship;

[0119] S24: Combining at least one of the reference viewpoint video frames in time order to form the reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system.

[0120] In an embodiment of the present application, at least one main viewpoint video frame can be obtained from a main viewpoint video frame sequence in a main viewpoint coordinate system, the main viewpoint video frame can be converted into a main viewpoint rotated video frame according to a rotation correspondence, the main viewpoint rotated video frame can be converted into a reference viewpoint video frame in an auxiliary viewpoint coordinate system according to a translation correspondence, and at least one reference viewpoint video frame can be combined in time sequence to form a reference viewpoint video frame sequence in an auxiliary viewpoint coordinate system. The rotation correspondence can be a rotation matrix, and the translation correspondence can be a translation matrix.

[0121] In the specific implementation, according to R 0,i and T 0,i , the frame M taken by the main viewpoint at time t t Convert to each auxiliary viewpoint, where the video frame converted to Viewi is used Indicates that Wherein i represents View i, i=1,2,...,n.

[0122] In the specific implementation, you can refer to Figure 4 Transform spatial relationships. Figure 4 , which shows a schematic diagram of a spatial relationship conversion process provided by an embodiment of the present application. When inter-frame coding is performed using the relationship between viewpoints, the video frame of View i at time t is The traditional method in the related art is to use the main viewpoint frame M t and auxiliary viewpoint frames However, due to the large difference in camera positions, the main viewpoint frame M t and auxiliary viewpoint frames The relative positions of some identical contents in the video frame may be far apart. In practical applications, due to the real-time requirements, the high-complexity full search algorithm is generally not used. Therefore, under the limitation of the search range, the motion estimation may fall into the local optimum, and the inter-frame prediction cannot make full use of the common information between different viewpoints. t Convert to Reference Viewpoint After that, the main viewpoint frame M is reflected t Reference viewpoint frame of the feature and The relative position distance of the same content in two video frames is small, so that the common information between viewpoints can be fully utilized to improve the coding efficiency.

[0123] The present application converts the main viewpoint video frame into the reference viewpoint video frame through the rotation correspondence and the translation correspondence, so as to ensure that the video frame can be accurately converted and encoded, thereby improving the efficiency and quality of multi-viewpoint video encoding; by converting the main viewpoint frame into the reference viewpoint frame, it is ensured that the relative position distance of the same content in the two video frames is small, and the common information between the viewpoints is fully utilized, thereby improving the encoding efficiency and prediction accuracy.

[0124] In an optional embodiment of the present application, the preset coding rule includes an inter-frame coding rule and an inter-view coding rule, and step S106 further includes the following sub-steps:

[0125] S31: Encode the main viewpoint video frame sequence using the inter-frame coding rule to obtain main viewpoint video coding information;

[0126] S32: Encode the auxiliary viewpoint video frame sequence by using the inter-frame coding rule, the inter-viewpoint coding rule, and the reference viewpoint video frame sequence to obtain auxiliary viewpoint video coding information;

[0127] S33: Determine the encoding information of the multi-view video according to the main view video encoding information and the auxiliary view video encoding information.

[0128] In an embodiment of the present application, the main viewpoint video frame sequence can be encoded using an inter-frame coding rule to obtain main viewpoint video coding information. The auxiliary viewpoint video frame sequence can be encoded using an inter-frame coding rule, an inter-viewpoint coding rule, and a reference viewpoint video frame sequence to obtain auxiliary viewpoint video coding information. The coding information of the multi-viewpoint video is determined based on the main viewpoint video coding information and the auxiliary viewpoint video coding information.

[0129] In the specific implementation, you can refer to Figure 5 Perform multi-view coding based on spatial relationship transformation. Figure 5 , showing a schematic diagram of a framework of multi-viewpoint coding based on spatial relationship transformation provided by an embodiment of the present application, Figure 5 In the , the main view corresponding to the main viewpoint View 0 can be encoded using HEVC, considering only the inter-frame correlation on the time axis. Similar to MV-HEVC, the auxiliary views corresponding to the auxiliary viewpoints View 1 and View 2 take into account both the inter-view and temporal correlations during inter-frame encoding. When utilizing the relationship between viewpoints, the present application first converts each frame in the main viewpoint View 0 to the corresponding viewpoint based on the rotation matrix and translation matrix between different viewpoints; and then uses the similarity between the converted reference viewpoint frames and the auxiliary viewpoint frames for inter-view encoding.

[0130] In an optional embodiment of the present application, step S32 further includes the following sub-steps:

[0131] S41: Encode the auxiliary viewpoint video frame sequence using the inter-frame coding rule to obtain auxiliary viewpoint first coding information;

[0132] S42: Encoding the auxiliary viewpoint video frame sequence using the inter-viewpoint coding rule and the reference viewpoint video frame sequence to obtain auxiliary viewpoint second coding information;

[0133] S43: Determine the auxiliary viewpoint video encoding information according to the auxiliary viewpoint first encoding information and the auxiliary viewpoint second encoding information.

[0134] In an embodiment of the present application, the auxiliary viewpoint video frame sequence can be encoded using an inter-frame coding rule to obtain auxiliary viewpoint first coding information, and then the auxiliary viewpoint video frame sequence can be encoded using an inter-viewpoint coding rule and a reference viewpoint video frame sequence to obtain auxiliary viewpoint second coding information. Finally, the auxiliary viewpoint video coding information can be determined based on the auxiliary viewpoint first coding information and the auxiliary viewpoint second coding information.

[0135] In the specific implementation, you can refer to Figure 5 , taking the B frame at time T2 of viewpoint 2 as an example, the B frame at time T2 of viewpoint 2 uses the P frame at time T0 of viewpoint 2 and the B frame at time T4 of viewpoint 2 on the time axis, and uses the B2 frame from viewpoint 0 to viewpoint 2 to reflect the spatial correspondence between viewpoints, where B2 = R 0,2 ×B+T 0,2 .

[0136] In an optional embodiment of the present application, the main viewpoint video and the auxiliary viewpoint video are collected within a preset period of time, and after converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, the method further includes the following steps:

[0137] S51: Acquire any one of the reference viewpoint video frames in the reference viewpoint video frame sequence;

[0138] S52: determining a region to be detected in the reference viewpoint video frame according to the spatial correspondence;

[0139] S53: setting a sliding detection window in the area to be detected; the size of the sliding detection window is determined by the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; the sliding detection window includes a number of pixels to be detected in the reference viewpoint video frame;

[0140] S54: in the process of the sliding detection window sliding in the area to be detected, obtaining the gray value of the pixel to be detected in the sliding detection window and the median of the gray values ​​of a number of the pixels to be detected in the sliding detection window;

[0141] S55: Determine the grayscale difference between the grayscale value of any pixel to be detected in the sliding detection window and the median of the grayscale values;

[0142] S56: If the grayscale difference is greater than a preset threshold, the pixel to be detected is replaced with a pixel corresponding to the median of the grayscale value;

[0143] S57: After the sliding detection window traverses the area to be detected, the reference viewpoint video frames formed after replacing the pixels to be detected are combined into a new reference viewpoint video frame sequence.

[0144] In the embodiment of the present application, any reference viewpoint video frame in the reference viewpoint video frame sequence is obtained, and the region to be detected in the reference viewpoint video frame is determined according to the spatial correspondence relationship, wherein the spatial correspondence relationship may be the above-mentioned rotation correspondence relationship and translation correspondence relationship.

[0145] Then, a sliding detection window can be set in the area to be detected, and the size of the sliding detection window is determined by the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera. The larger the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, the larger the size of the sliding detection window, and the smaller the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, the smaller the size of the sliding detection window.

[0146] The sliding detection window includes several pixels to be detected in the reference viewpoint video frame. When the sliding detection window slides in the area to be detected, the grayscale value of the pixel to be detected in the sliding detection window and the median grayscale value of several pixels to be detected in the sliding detection window can be obtained, and the grayscale difference between the grayscale value of any pixel to be detected in the sliding detection window and the median grayscale value is determined. If the grayscale difference is greater than a preset threshold, the pixel to be detected is replaced with the pixel corresponding to the median grayscale value.

[0147] After the sliding detection window traverses the area to be detected, the reference viewpoint video frames formed after replacing the pixels to be detected can be combined into a new reference viewpoint video frame sequence.

[0148] In the specific implementation, you can refer to Figure 6 Get the angles of different viewpoint cameras, refer to Figure 6, shows a schematic diagram of different viewpoint camera shooting angles provided by an embodiment of the present application. Due to the different shooting angles of different viewpoint cameras, the content captured by each viewpoint is different. For example, the camera of the auxiliary viewpoint View 2 is placed on the right side of the target object, and the camera of the main viewpoint View 0 is placed in the middle part of the target object. The video frame under the viewpoint View 2 can obtain more information on the right side of the target object, while the video frame under the viewpoint View 0 may lose some detail information on the right side of the target object. Therefore, when the video frame of the main viewpoint View 0 is converted to the auxiliary viewpoint View 2, there may be a loss of detail information.

[0149] In the specific implementation, you can refer to Figure 7 Get the missing details diagram, refer to Figure 7 , which shows a schematic diagram of video frame detail information loss provided by an embodiment of the present application, Figure 7 In the area within the black frame, detailed information in the image is missing.

[0150] In the specific implementation, in the converted main view video frame, the grayscale value of the part where the detail information is lost is significantly different from the grayscale value of its adjacent pixels. And when the main view video frame is converted to the right, the lost information is often concentrated on the right, and vice versa. Therefore, when detecting the information loss part, this patent proposes an adaptive area sliding window detail loss detection method. First, the candidate area is calculated according to the rotation matrix and the translation matrix, that is, Area = computeArea (R 0,i ,T 0,i ), it can be understood that if the auxiliary viewpoint is to the right of the main viewpoint, the right half of the video frame is used as the adaptive region. By selecting the candidate region, it is possible to avoid searching for the missing details in the full image frame, which can improve the retrieval efficiency.

[0151] Then a k×k sliding window is set, and the window slides in the candidate area to traverse the entire candidate area. During the sliding process, each pixel is compared with the median gray value of the pixel in the window. If the difference between a certain pixel and the median gray value of the pixel in the window is greater than the set threshold, the pixel is regarded as a missing part and filled with the pixel where the median is located. By filling in the details, the missing part of the converted image can be replaced by the median of the adjacent pixels, which is closer to the pixel corresponding to the auxiliary viewpoint, so that the redundancy between different viewpoint frames can be fully utilized to improve the coding efficiency.

[0152] In an optional embodiment of the present application, step S54 includes the following sub-steps:

[0153] S61: in the process of the sliding detection window sliding in the area to be detected, obtaining in real time the grayscale values ​​of a number of pixels to be detected in the sliding detection window;

[0154] S62: sorting the grayscale values ​​of the pixels to be detected in ascending order;

[0155] S63: Taking the median of the sorted grayscale values ​​as the median of the grayscale values ​​of the pixels to be detected.

[0156] In an embodiment of the present application, while the sliding detection window is sliding in the area to be detected, the grayscale values ​​of several pixels to be detected in the sliding detection window can be obtained in real time, the grayscale values ​​of the several pixels to be detected can be sorted in ascending order, and the median of the sorted grayscale values ​​can be used as the median of the grayscale values ​​of the several pixels to be detected.

[0157] In a specific implementation, the grayscale values ​​of all pixels in each window position can be extracted. Then the extracted grayscale values ​​are sorted. According to the window size, the median of the sorted grayscale values ​​is calculated. For an odd-sized window, the median is the middle value after sorting; for an even-sized window, the median can be the average of the two middle values.

[0158] Reference Figure 8 , shows a schematic diagram of a framework of a multi-view video encoding method combining spatial relationship conversion and detail filling provided by an embodiment of the present application. The specific contents are as follows:

[0159] The multi-view video encoding method combining spatial relationship conversion and detail filling mainly includes two parts: the first part is spatial relationship conversion, which first obtains the spatial relationship between cameras and then performs viewpoint conversion based on the spatial relationship; the second part is sliding window detail filling in the adaptive area. For the converted video frames, a sliding window is set and the sliding area is adaptively adjusted based on the spatial relationship to improve the efficiency of detail filling.

[0160] Reference Fig. 9 , shows a schematic structural diagram of a multi-view video encoding device provided by an embodiment of the present application, wherein the multi-view video includes a main view video captured by a main view camera and an auxiliary view video captured by an auxiliary view camera, and the device includes:

[0161] An installation position acquisition module 901 is used to acquire the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0162] A coordinate system establishing module 902, used to establish a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and to establish an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera;

[0163] A spatial correspondence determination module 903, configured to determine a spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera;

[0164] A video frame sequence acquisition module 904 is used to acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video;

[0165] A spatial conversion module 905, configured to convert the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence;

[0166] The encoding module 906 is used to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence by using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

[0167] Optionally, the main viewpoint camera and the auxiliary viewpoint camera are set in a preset scene, the spatial correspondence includes a rotation correspondence and / or a translation correspondence, and the spatial correspondence determination module 903 includes:

[0168] A common feature point acquisition module, used to select at least one common feature point in the preset scene;

[0169] A feature point main view coordinate acquisition module, used to determine the main view feature coordinates of the common feature point in the main view coordinate system according to the spatial relationship between the installation position of the main view camera and the common feature point;

[0170] A feature point auxiliary view coordinate acquisition module, used to determine the auxiliary view feature coordinates of the common feature point in the auxiliary view coordinate system according to the spatial relationship between the installation position of the auxiliary view camera and the common feature point;

[0171] A matching point pair acquisition module, used for taking the main viewpoint feature coordinates and the auxiliary viewpoint feature coordinates as the matching point pair of the common feature points;

[0172] A rotation and translation relationship acquisition module is used to determine the rotation correspondence relationship and / or translation correspondence relationship between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the matching point pair.

[0173] Optionally, the video frame sequence acquisition module 904 includes:

[0174] A main viewpoint video frame acquisition module, used to acquire at least one main viewpoint video frame from the main viewpoint video frame sequence in the main viewpoint coordinate system;

[0175] A main viewpoint video frame rotation module, used for converting the main viewpoint video frame into a main viewpoint rotated video frame according to the rotation correspondence;

[0176] A main viewpoint video frame translation module, used for converting the main viewpoint rotated video frame into a reference viewpoint video frame in the auxiliary viewpoint coordinate system according to the translation correspondence;

[0177] The video frame sequence acquisition submodule is used to combine at least one of the reference viewpoint video frames in time sequence to form the reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system.

[0178] Optionally, the preset coding rule includes an inter-frame coding rule and an inter-view coding rule, and the coding module 906 includes:

[0179] A main coding information acquisition module, used to encode the main viewpoint video frame sequence using the inter-frame coding rule to obtain main viewpoint video coding information;

[0180] an auxiliary coding information acquisition module, configured to encode the auxiliary viewpoint video frame sequence by adopting the inter-frame coding rule, the inter-viewpoint coding rule, and the reference viewpoint video frame sequence to obtain auxiliary viewpoint video coding information;

[0181] The video encoding information acquisition module is used to determine the encoding information of the multi-view video according to the main view video encoding information and the auxiliary view video encoding information.

[0182] Optionally, the auxiliary coding information acquisition module includes:

[0183] A first coding information acquisition module, configured to encode the auxiliary viewpoint video frame sequence using the inter-frame coding rule to obtain auxiliary viewpoint first coding information;

[0184] A second coding information acquisition module, configured to encode the auxiliary viewpoint video frame sequence by adopting the inter-viewpoint coding rule and the reference viewpoint video frame sequence to obtain auxiliary viewpoint second coding information;

[0185] The auxiliary coding information acquisition submodule is used to determine the auxiliary viewpoint video coding information according to the auxiliary viewpoint first coding information and the auxiliary viewpoint second coding information.

[0186] Optionally, the main viewpoint video and the auxiliary viewpoint video are collected within a preset time period, and after converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, the apparatus further comprises:

[0187] A reference viewpoint video frame acquisition module, used to acquire any reference viewpoint video frame in the reference viewpoint video frame sequence;

[0188] A module for acquiring an area to be detected, used to determine the area to be detected in the reference viewpoint video frame according to the spatial correspondence;

[0189] A sliding window acquisition module, used for setting a sliding detection window in the area to be detected; the size of the sliding detection window is determined by the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; the sliding detection window includes a number of pixels to be detected in the reference viewpoint video frame;

[0190] A gray value acquisition module, used for acquiring the gray value of the pixel to be detected in the sliding detection window and the median of the gray values ​​of several pixels to be detected in the sliding detection window during the sliding detection window sliding in the area to be detected;

[0191] A grayscale value comparison module, used to determine the grayscale difference between the grayscale value of any pixel to be detected in the sliding detection window and the median of the grayscale values;

[0192] A pixel replacement module, used to replace the pixel to be detected with a pixel corresponding to the median of the grayscale value if the grayscale difference is greater than a preset threshold;

[0193] The reference viewpoint video frame optimization module is used to form a new reference viewpoint video frame sequence by replacing the reference viewpoint video frames formed after the sliding detection window traverses the area to be detected.

[0194] Optionally, the gray value acquisition module includes:

[0195] A gray value acquisition submodule, used for acquiring the gray values ​​of a number of pixels to be detected in the sliding detection window in real time during the sliding detection window sliding in the area to be detected;

[0196] A gray value sorting module, used to sort the gray values ​​of the pixels to be detected in ascending order;

[0197] The grayscale median acquisition module is used to use the median of the sorted grayscale values ​​as the grayscale median of the plurality of pixels to be detected.

[0198] An embodiment of the present application also provides an electronic device, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, it implements the multi-viewpoint video encoding method as described above.

[0199] An embodiment of the present application further provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the multi-view video encoding method as described above is implemented.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0201] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0203] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0207] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0208] The above is a detailed introduction to the provided multi-viewpoint video encoding method, device, electronic device and medium. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for encoding a multi-view video, characterized in that: The multi-viewpoint video includes a main viewpoint video captured by a main viewpoint camera and an auxiliary viewpoint video captured by an auxiliary viewpoint camera, and the method includes: Obtaining the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; Establishing a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and establishing an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera; Determining a spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; Acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video; Converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence; The main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence are encoded using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

2. The method according to claim 1, characterized in that The main viewpoint camera and the auxiliary viewpoint camera are arranged in a preset scene, the spatial correspondence relationship includes a rotation correspondence relationship and / or a translation correspondence relationship, and the spatial correspondence relationship between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera, including: Selecting at least one common feature point in the preset scene; Determining the main viewpoint feature coordinates of the common feature point in the main viewpoint coordinate system according to the spatial relationship between the installation position of the main viewpoint camera and the common feature point; Determining the auxiliary viewpoint feature coordinates of the common feature point in the auxiliary viewpoint coordinate system according to the spatial relationship between the installation position of the auxiliary viewpoint camera and the common feature point; Taking the main viewpoint feature coordinates and the auxiliary viewpoint feature coordinates as a matching point pair of the common feature points; The rotation correspondence and / or translation correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system is determined according to the matching point pair.

3. The method according to claim 2, characterized in that The converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence includes: Acquire at least one main viewpoint video frame from the main viewpoint video frame sequence in the main viewpoint coordinate system; Converting the main viewpoint video frame into a main viewpoint rotated video frame according to the rotation correspondence; Converting the main viewpoint rotated video frame into a reference viewpoint video frame in the auxiliary viewpoint coordinate system according to the translation correspondence; At least one of the reference viewpoint video frames is combined in time sequence to form the reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system.

4. The method according to claim 3, characterized in that The preset coding rule includes an inter-frame coding rule and an inter-view coding rule, and the encoding of the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence by using the preset coding rule and the reference viewpoint video frame sequence to obtain the coding information of the multi-viewpoint video includes: Encoding the main viewpoint video frame sequence using the inter-frame coding rule to obtain main viewpoint video coding information; Encoding the auxiliary viewpoint video frame sequence by using the inter-frame coding rule, the inter-viewpoint coding rule, and the reference viewpoint video frame sequence to obtain auxiliary viewpoint video coding information; The encoding information of the multi-view video is determined according to the main view video encoding information and the auxiliary view video encoding information.

5. The method according to claim 4, characterized in that The adopting the inter-frame coding rule, the inter-view coding rule, and the reference viewpoint video frame sequence to encode the auxiliary viewpoint video frame sequence to obtain auxiliary viewpoint video coding information includes: Encoding the auxiliary viewpoint video frame sequence using the inter-frame coding rule to obtain auxiliary viewpoint first coding information; Encoding the auxiliary viewpoint video frame sequence using the inter-viewpoint coding rule and the reference viewpoint video frame sequence to obtain auxiliary viewpoint second coding information; The auxiliary viewpoint video encoding information is determined according to the auxiliary viewpoint first encoding information and the auxiliary viewpoint second encoding information.

6. The method according to claim 1, characterized in that The main viewpoint video and the auxiliary viewpoint video are collected within a preset period of time. After converting the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence, the method further includes: Acquire any one of the reference viewpoint video frames in the reference viewpoint video frame sequence; Determine the area to be detected in the reference viewpoint video frame according to the spatial correspondence; A sliding detection window is set in the area to be detected; the size of the sliding detection window is determined by the distance between the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; the sliding detection window includes a number of pixels to be detected in the reference viewpoint video frame; In the process of the sliding detection window sliding in the area to be detected, obtaining the gray value of the pixel to be detected in the sliding detection window and the median of the gray values ​​of several pixels to be detected in the sliding detection window; Determine a grayscale difference between a grayscale value of any pixel to be detected in the sliding detection window and a median of the grayscale values; If the grayscale difference is greater than a preset threshold, the pixel to be detected is replaced with a pixel corresponding to the median of the grayscale value; After the sliding detection window traverses the area to be detected, the reference viewpoint video frames formed after replacing the pixels to be detected are formed into a new reference viewpoint video frame sequence.

7. The method according to claim 6, characterized in that In the process of the sliding detection window sliding in the area to be detected, obtaining the grayscale value of the pixel to be detected in the sliding detection window and the median of the grayscale values ​​of several pixels to be detected in the sliding detection window includes: In the process of the sliding detection window sliding in the area to be detected, grayscale values ​​of a number of pixels to be detected in the sliding detection window are acquired in real time; Sorting the grayscale values ​​of the pixels to be detected in ascending order; The median of the sorted grayscale values ​​is used as the median of the grayscale values ​​of the pixels to be detected.

8. A multi-view video encoding device, characterized in that: The multi-viewpoint video includes a main viewpoint video collected by a main viewpoint camera and an auxiliary viewpoint video collected by an auxiliary viewpoint camera, and the device includes: An installation position acquisition module, used to acquire the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; A coordinate system establishment module, used to establish a main viewpoint coordinate system according to the installation position of the main viewpoint camera, and to establish an auxiliary viewpoint coordinate system according to the installation position of the auxiliary viewpoint camera; A spatial correspondence determination module, used to determine the spatial correspondence between the main viewpoint coordinate system and the auxiliary viewpoint coordinate system according to the installation position of the main viewpoint camera and the installation position of the auxiliary viewpoint camera; A video frame sequence acquisition module, used to acquire a main viewpoint video frame sequence in the main viewpoint video and an auxiliary viewpoint video frame sequence in the auxiliary viewpoint video; A spatial conversion module, configured to convert the main viewpoint video frame sequence in the main viewpoint coordinate system into a reference viewpoint video frame sequence in the auxiliary viewpoint coordinate system according to the spatial correspondence; The encoding module is used to encode the main viewpoint video frame sequence and the auxiliary viewpoint video frame sequence by using a preset encoding rule and the reference viewpoint video frame sequence to obtain encoding information of the multi-viewpoint video.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the multi-view video encoding method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-view video encoding method according to any one of claims 1 to 7 is implemented.

Images