A method, apparatus, server, and storage medium for processing video streams.

Abstract

This disclosure proposes a video stream processing method, apparatus, server, and storage medium. The method includes: acquiring a scene image including a real screen captured by a first image acquisition device, and generating a virtual screen corresponding to the real screen from a first perspective; mapping the video frame of a video stream to be played onto the virtual screen from the first perspective to generate a first video stream from the first perspective; converting the virtual screen from the first perspective into a virtual screen from a second perspective; mapping the video frame of the video stream to be played onto the virtual screen from the second perspective to generate a second video stream from the second perspective; and feeding back the first video stream and the second video stream to a virtual reality device. One or more embodiments of this disclosure provide a multi-view video viewing method for users.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a video stream processing method and device, a server, and a storage medium. BACKGROUND

[0002] At present, virtual reality (VR) technology is widely applied in online concerts, online events, and virtual reality roaming, etc. due to its ability to provide panoramic videos and better immersion. By cooperating with a virtual reality device (e.g., a VR headset), a user can be provided with an immersive viewing experience.

[0003] By deploying a panoramic camera in an offline scene and then integrating the video pictures taken by the panoramic camera into a panoramic video stream, the panoramic video stream can be sent to a virtual reality device of a user. In this way, the user can watch the offline scene online with the help of the virtual reality device.

[0004] However, the existing panoramic video stream processing method can only provide a single-viewing mode for users, which greatly limits video watching. SUMMARY

[0005] Therefore, one or more embodiments of the present disclosure provide a video stream processing method and device, a server, and a storage medium, which can provide a multi-viewing mode for users.

[0006] In one aspect, the present disclosure provides a video stream processing method, which includes: obtaining a scene image containing a real screen taken by a first image acquisition device, and generating a virtual screen corresponding to the real screen in a first view; mapping a video picture of a to-be-broadcast video stream onto the virtual screen in the first view, and generating a first video stream in the first view according to a first mapping picture on the virtual screen in the first view; converting the virtual screen in the first view into a virtual screen in a second view according to a relative position between the first image acquisition device and a second image acquisition device; mapping the video picture of the to-be-broadcast video stream onto the virtual screen in the second view, and generating a second video stream in the second view according to a second mapping picture on the virtual screen in the second view; and feeding back the first video stream and the second video stream to a virtual reality device, so that the virtual reality device plays the first video stream or the second video stream in different views.

[0007] This disclosure also provides a video stream processing apparatus, the apparatus comprising: a first viewpoint processing unit, configured to acquire a scene image including a real screen captured by a first image acquisition device, and generate a virtual screen corresponding to the real screen from a first viewpoint; a first mapping unit, configured to map the video frame of the video stream to be played onto the virtual screen from the first viewpoint, and generate a first video stream from the first viewpoint based on a first mapped image on the virtual screen from the first viewpoint; a viewpoint conversion unit, configured to convert the virtual screen from the first viewpoint to a virtual screen from a second viewpoint based on the relative position between the first image acquisition device and the second image acquisition device; a second mapping unit, configured to map the video frame of the video stream to be played onto the virtual screen from the second viewpoint, and generate a second video stream from the second viewpoint based on a second mapped image on the virtual screen from the second viewpoint; and a video stream feedback unit, configured to feed back the first video stream and the second video stream to a virtual reality device, so that the virtual reality device plays the first video stream or the second video stream from different viewpoints.

[0008] This disclosure also provides a server, which includes a memory and a processor, the memory for storing a computer program that, when executed by the processor, implements the video stream processing method described above.

[0009] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described video stream processing method.

[0010] The technical solution provided by one or more embodiments of this disclosure can first capture scene images including a real screen using a first image acquisition device. Considering that the size of the real screen may be small, in order to improve the user's viewing experience, a virtual screen corresponding to the real screen can be generated from a first-view perspective. By mapping the video frame of the video stream to be played onto the virtual screen, a first video stream from the first-view perspective can be formed. This first video stream can then be fed back to the user's virtual reality device, allowing the user to view the first video stream from the first-view perspective. To provide a multi-view video viewing method, a second image acquisition device can also be deployed in the offline scene. Then, based on the relative position between the first and second image acquisition devices, the virtual screen from the first-view perspective can be converted into a virtual screen from the second-view perspective. Then, by mapping the video frame of the video stream to be played onto the virtual screen from the second-view perspective, a second video stream from the second-view perspective can be obtained, which can also be fed back to the user's virtual reality device. In this way, by providing users with video streams from different perspectives, users can choose a suitable viewing angle independently, thereby improving the user's video viewing experience. Attached Figure Description

[0011] The features and advantages of the embodiments of this disclosure will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the present disclosure in any way. In the drawings:

[0012] Figure 1 This disclosure illustrates an offline scenario combined with a virtual screen;

[0013] Figure 2 A schematic diagram of the steps of a video stream processing method according to one embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram illustrating the generation of a virtual screen in one embodiment of this disclosure is shown;

[0015] Figure 4 A schematic diagram of virtual screen rotation is shown in one embodiment of this disclosure;

[0016] Figure 5 (a) shows a schematic diagram of a fisheye image in a specific application scenario of this disclosure;

[0017] Figure 5 (b) shows a schematic diagram of a corrected image in a specific application scenario of this disclosure;

[0018] Figure 6 A schematic diagram of the functional modules of a video stream processing apparatus according to one embodiment of the present disclosure is shown;

[0019] Figure 7A schematic diagram of the server structure in one embodiment of this disclosure is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] Please see Figure 1 In offline scenarios, physical screens are typically deployed. Depending on the scale of the offline scenario, the number of physical screens can be one or more. This disclosure uses a single physical screen as an example to illustrate the solution. The video stream to be played can be displayed on the physical screen, and the audience in the offline scenario can watch the video on the physical screen from their seats. Currently, in order to allow some audiences who cannot attend the offline scenario to experience the atmosphere of being there in person, VR technology can be used to film the offline scenario and ultimately produce a video stream. This video stream can be fed back to the user's virtual reality device. In this way, users can experience the feeling of being there by watching the video stream played on their virtual reality device.

[0022] Currently, in offline scenarios, a panoramic camera is typically deployed in a location with a relatively open field of view. For example, in Figure 1 In this system, the panoramic camera can be positioned in the center of the offline scene (a central circle). This camera captures a 360-degree view of the offline scene and creates a panoramic video stream. When a user watches the panoramic video stream through a virtual reality device, it's as if they are viewing the entire offline scene from the perspective of this panoramic camera. However, users sometimes need to change their viewing angle while watching the panoramic video stream. For example, in... Figure 1 In this scenario, a user might want to switch from the view of the central panoramic camera to a view from the upper left, but based on the existing solution, this view switching is not possible. Furthermore, when this view switching requirement arises, another issue needs to be addressed: when viewing the offline scene from the central panoramic camera's perspective, the actual screen is directly in front of the user, providing a good viewing experience. However, switching from the central panoramic camera's view to the upper left results in the user not being able to directly face the actual screen, thus degrading the viewing experience.

[0023] In view of this, one embodiment of this disclosure provides a video stream processing method. This method can be applied to a server providing panoramic video streams. The server can communicate with various cameras in an offline scene to acquire scene images captured by each camera. The server processes the acquired scene images to form a multi-view video stream and feeds this multi-view video stream back to the virtual reality device of an online user. Please refer to [link to relevant documentation]. Figure 2 This method may include the following steps.

[0024] S1: Acquire a scene image containing a real screen captured by the first image acquisition device, and generate a virtual screen corresponding to the real screen from the first perspective.

[0025] In this embodiment, a first image acquisition device can be deployed in an offline scenario. This first image acquisition device can be a camera or other device with image capture capabilities. For ease of description, a first camera will be used as an example to describe the first image acquisition device. Typically, the first camera can be deployed facing the actual screen. For example, the first camera could be... Figure 1 A central panoramic camera. And to ensure a better viewing experience, this first camera is typically a 360-degree panoramic camera. Of course, depending on the application scenario, the position and type of the first camera can be flexibly adjusted, and this disclosure does not impose any limitations on this.

[0026] Furthermore, to create multi-view video streams, one or more second image acquisition devices can be deployed in the offline scene. For ease of description, the second camera will be used as an example to describe the second image acquisition device below. For example, in Figure 1 In this configuration, three additional second cameras (in a triangle) can be deployed, each corresponding to a different viewing angle. The deployment location and type of the second cameras can also be flexibly adjusted according to actual application requirements, for example, in... Figure 1 In the specific application example shown, the three second cameras are located at three different locations around the offline venue, and all three second cameras can be 180-degree panoramic cameras.

[0027] In this embodiment, the first camera can capture scene images including the real screen, and the format of these scene images is related to the type of the first camera. For example, in a practical application scenario, the first camera could be a fisheye camera, and the captured scene image could be a fisheye image. The first camera can correspond to a first viewpoint, so the scene image captured by the first camera can be used as an image viewed from that first viewpoint. Considering that in some scenarios the size of the real screen may be relatively small, to improve the viewing experience for online users, a larger virtual screen can be simulated based on the location of the real screen. Generally, the area occupied by the real screen can be identified from the scene image, and then this area can be enlarged to obtain the area corresponding to the virtual screen.

[0028] In one implementation, corner detection algorithms, such as the Kitchen-Rosenfeld corner detection algorithm, the Harris corner detection algorithm, the KLT corner detection algorithm, and the SUSAN corner detection algorithm, can be used to identify the four real corners of the actual screen from the scene image. Then, by magnification, virtual corners corresponding to each real intersection point are generated. For example, in... Figure 3 In this method, four real corner points can be extended outwards along the diagonal of the real screen by a certain distance, thus obtaining four virtual corner points. The area formed by these multiple virtual corner points can then serve as the virtual screen in the first-person perspective. Therefore, by simulating the real screen, a virtual screen with a flexible and adjustable size in the first-person perspective can be obtained.

[0029] It should be noted that in practical applications, the number of virtual screens generated in a first-person perspective can be more than one. Following the method described above, a first virtual screen corresponding to the real screen can be generated at the location of the real screen. When viewed from a first-person perspective, this first virtual screen can be directly facing the viewer, providing a good viewing experience. However, if the perspective is switched to that of a second camera (e.g., ...), the virtual screen will not function correctly. Figure 1 If the viewpoint is from the top left corner, then the first virtual screen is no longer directly facing the viewer, thus affecting the viewer's viewing experience. Therefore, in one embodiment, after generating the first virtual screen at the location of the real screen, separate virtual screens facing each second camera can be generated. Specifically, using the central axis of the second camera's shooting range as a reference, a second virtual screen corresponding to the real screen can be generated along the direction of that central axis. For example, in... Figure 4 In this process, the central axis of the second camera's shooting range can be determined, as well as the perpendicular line between the first camera and the first virtual screen. Then, the angle between the central axis and the perpendicular line can be calculated. Figure 4In this scenario, the angle between the first camera and the second camera in the upper left corner can be 120 degrees. Therefore, when generating the second virtual screen, it can be directly rotated 120 degrees clockwise around the first camera in the camera coordinate system. Similarly, for second cameras in other positions, the corresponding second virtual screen can be obtained by adaptively rotating the first virtual screen. In this way, after generating one or more second virtual screens, the first and second virtual screens can be used as virtual screens generated in the first-person perspective.

[0030] In practical applications, the specific location information of both the first and second virtual screens can be represented by coordinates in the camera coordinate system of the first camera. These coordinates could be, for example, the coordinates of the four corner points of each virtual screen in the camera coordinate system of the first camera.

[0031] In this way, when the viewpoint changes, there will always be a virtual screen facing the new viewpoint, thus ensuring that the user's viewing experience is not affected.

[0032] In practical applications, since the scene image captured by the first camera may have some distortion, in order to ensure the accuracy of the recognition results when identifying the corner points of the real screen from the scene image, the scene image can be dedistorted first, and then the corner points of the real screen can be identified from the dedistorted image.

[0033] Specifically, in one embodiment, a calibration model for the first camera can be obtained based on the type of the first camera. For example, when the first camera is a fisheye camera, the corresponding calibration model can be a KBCM model. After determining the calibration model of the first camera, the calibration coefficients of the first camera can be obtained through this calibration model. In practical applications, these calibration coefficients may include multiple intrinsic parameters and multiple distortion coefficients.

[0034] In this embodiment, after obtaining the calibration coefficients of the first camera, a projection function can be determined using these coefficients. This projection function can correct the distortion of the scene image, thereby converting the scene image into a corrected image. Taking a fisheye camera as an example, the fisheye image captured by the fisheye camera can be as follows: Figure 5 As shown in (a), Figure 5 In (a), the edges of the checkerboard pattern will be curved. After distortion correction by calibration coefficients, the result can be obtained. Figure 5(b) shows the corrected image. In this corrected image, the edges of the checkerboard pattern are restored to normal. This checkerboard pattern can represent the real screen. After converting the scene image into a corrected image, a region containing the real screen can be cropped from the corrected image, and multiple real corner points of the real screen can be identified in this region. The purpose of cropping is to reduce the influence of other content in the corrected image on corner detection, thereby improving the accuracy of corner detection.

[0035] In a specific application example, the projection function π(x,i) determined by the KBCM model can be expressed as:

[0036]

[0037]

[0038] θ = atan2(r, z),

[0039] d(θ)=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9

[0040] Where vector x represents a point in the fisheye image, (x, y, z) represents the coordinates of that point, and f x f y c x c y The four intrinsic parameters are k1, k2, k3, and k4, which are the four distortion coefficients.

[0041] S3: Map the video frame of the video stream to be played onto the virtual screen under the first viewpoint, and generate the first video stream under the first viewpoint based on the first mapped image on the virtual screen under the first viewpoint.

[0042] In this embodiment, after simulating a virtual screen, the video frame of the video stream to be played needs to be mapped onto the virtual screen so that the user can view the video frame of the video stream through the virtual reality device. The size of the video frame in the video stream to be played usually does not match the virtual screen, and the virtual screen cannot adaptively adjust the video frame like a real screen. Therefore, in this embodiment, the video frame can be mapped onto the virtual screen to obtain a video frame that matches the size of the virtual screen.

[0043] Specifically, in the camera coordinate system of the first camera, the mapping process between the video frame and the virtual screen can be equivalent to solving the homography matrix between the video frame and the virtual screen. This homography matrix can then serve as the mapping matrix between the video frame and the virtual screen. When solving the mapping matrix, multiple sets of mapping points can be determined between the video frame and the virtual screen. For example, four corner points of the video frame and four corner points of the virtual screen can be selected, and the corresponding corner points can form a set of mapping points. For instance, the top-left corner point of the video frame can be mapped to the top-left corner point of the virtual screen. The bottom-right corner point of the video frame can be mapped to the bottom-right corner point of the virtual screen, and so on. After determining multiple sets of mapping points, a mapping matrix can be generated based on the position information of these mapping points in the camera coordinate system of the first camera. The position information can be represented by coordinate values ​​in the camera coordinate system of the first camera.

[0044] In a specific application scenario, the formula for calculating the mapping matrix can be as follows:

[0045]

[0046] in, It can represent the x-coordinate of the j-th point in the i-th image. Let h represent the ordinate of the j-th point in the i-th image. The video frame can be used as the first image, and the virtual screen as the second image. Each image uses four points. Finally, the solution obtained from h... i The resulting matrix can then be used as a mapping matrix.

[0047] In this embodiment, after generating the mapping matrix, each pixel in the video frame of the video stream to be played can be mapped onto the virtual screen in the first-person perspective, thereby completing the video frame mapping process.

[0048] It should be noted that if there are multiple virtual screens in the first-person perspective, the above video image mapping process can be performed on each virtual screen to ensure that each virtual screen has a mapped video image. For example, the video image of the video stream to be played can be mapped to the first virtual screen and the second virtual screen in the aforementioned steps, respectively.

[0049] In this embodiment, after the video image is mapped onto the virtual screen, a processed first mapped image can be obtained. Each first mapped image can be regarded as a processed video image. In this way, the first mapped images can be sorted to form a first video stream from the first perspective.

[0050] As can be seen, after the above processing, the video stream to be played on the real screen can be processed into the first video stream in the first-person perspective. This first video stream corresponds to the video screen on the virtual screen. Therefore, after the first video stream is pushed to the user's virtual reality device, the user will only see the video screen on the virtual screen and will not see the video screen on the real screen.

[0051] In practical applications, after the above-described video image mapping steps, the resulting first mapped image is typically a planar image, while the video stream ultimately provided to the user's virtual reality device usually displays a panoramic image. Therefore, after obtaining the first mapped image on the virtual screen, each first mapped image can be inversely projected into a corresponding panoramic image based on the calibration coefficients of the first camera. Subsequently, the video stream composed of these panoramic images can be used as the first video stream from the first perspective. The process of inversely projecting the first mapped image can be viewed as solving for the inverse matrix of the projection function π(x,i). Knowing the intrinsic parameters of the first camera, this inverse matrix can be easily solved. Specifically, in an application example, this inverse matrix can be expressed as:

[0052]

[0053]

[0054]

[0055]

[0056] θ*=d -1 (r u ),

[0057] Here, u and v can represent the x and y coordinates of a pixel in the first mapped image. In this way, the inverse matrix of the projection function can be solved by the coordinate values ​​and intrinsic parameters, and then the first mapped image can be inversely projected into the corresponding panoramic image.

[0058] S5: Based on the relative position between the first image acquisition device and the second image acquisition device, convert the virtual screen under the first viewpoint into a virtual screen under the second viewpoint.

[0059] In this embodiment, the second camera can correspond to the second viewpoint. After obtaining the virtual screen under the first viewpoint, the virtual screen under the second viewpoint can be conveniently obtained according to the relative position between the first camera and the second camera.

[0060] Specifically, the position of the virtual screen in the first-view perspective can be represented by coordinate values ​​in the camera coordinate system of the first camera. Therefore, by transforming the coordinates from the first camera's camera coordinate system to the world coordinate system, and then from the world coordinate system to the second camera's camera coordinate system, the position of the virtual screen in the first-view perspective can be easily transformed to the second-view perspective, thus obtaining the virtual screen in the second-view perspective. In practical applications, the coordinate transformation between the camera coordinate system and the world coordinate system requires the use of the camera's extrinsic parameters. Therefore, based on the extrinsic parameters of the first camera, the virtual screen in the first-view perspective can be mapped to the world coordinate system; then, based on the extrinsic parameters of the second camera, the virtual screen in the world coordinate system can be mapped to the camera coordinate system of the second camera, thus forming the virtual screen in the second-view perspective. The specific conversion process is a mature technology in this field and will not be elaborated here.

[0061] S7: Map the video frame of the video stream to be played onto the virtual screen in the second perspective, and generate the second video stream in the second perspective based on the second mapped image on the virtual screen in the second perspective.

[0062] In this embodiment, after obtaining the virtual screen from the second perspective, the video frame of the video stream to be played can be mapped onto the virtual screen from the second perspective, as described in step S3. Finally, a second video stream from the second perspective can be generated based on the second mapped image on the virtual screen. For the specific implementation process, please refer to the description of the preceding steps; it will not be repeated here.

[0063] S9: Feed back the first video stream and the second video stream to the virtual reality device so that the virtual reality device can play the first video stream or the second video stream from different perspectives.

[0064] In this embodiment, after obtaining the first video stream and the second video stream from different perspectives, both the first video stream and the second video stream can be fed back to the virtual reality device of the online user. The online user can select the corresponding video stream to play on the virtual reality device and can switch between different video streams, thereby watching the video footage of the offline scene from different perspectives.

[0065] In one implementation, considering that if the final video stream is formed solely from the video footage mapped onto a virtual screen, online users would only see the video footage on the virtual screen and would not be able to view the offline environment, this implementation can fuse the video footage mapped onto the virtual screen with real-world footage of the offline scene captured by a camera, thereby obtaining a video stream that combines both video footage and the offline environment.

[0066] Specifically, taking the first camera as an example, it can capture scene images of the offline environment. These images may include the layout of the offline scene, the audience, and other real-world elements. The video feed from the first video stream can then be fused with these scene images to obtain a fused image that combines the virtual screen and the offline reality. This fused image stream can then serve as the first fused video stream from a first-person perspective. The same principle applies to the second camera; its video feed can be fused with the scene images captured by the second camera to generate a second fused video stream from a second-person perspective. After generating the first and second fused video streams, these streams can be fed back to the online user's virtual reality device. In this way, online users can not only view the video feed on the virtual screen from different perspectives but also see the real-world offline scene from different viewpoints, greatly enhancing their sense of immersion.

[0067] In one implementation, when fusing video footage from a virtual screen with scene images captured by a camera, a preset fusion coefficient can be determined based on the desired display effect. This preset fusion coefficient can be a real number between 0 and 1. Then, for any color component in the video footage and scene image, the component data of that color component can be weighted and summed based on the preset fusion coefficient to generate the fused data corresponding to that color component. Finally, the image composed of the fused data of each color component can be used as the fused image.

[0068] For example, if both video footage and scene images are RGB images, then the color components can be divided into R, G, and B components. The fused data of these three color components can be calculated using the following formula:

[0069] C_r = A_r * alpha + B_r * (1 - alpha)

[0070] C_g = A_g*alpha + B_g*(1-alpha)

[0071] C_b = A_b * alpha + B_b * (1 - alpha)

[0072] Where alpha represents the preset fusion coefficient, C_r represents the fused R component data, C_g represents the fused G component data, C_b represents the fused B component data, A_r and B_r represent the R component data of the video frame and the scene image respectively, A_g and B_g represent the G component data of the video frame and the scene image respectively, and A_b and B_b represent the B component data of the video frame and the scene image respectively.

[0073] It should be noted that when there are multiple virtual screens, different types of operations can be performed on each virtual screen. For example, the video frame of the video stream to be played can be mapped to the first virtual screen and the second virtual screen respectively, and both the first virtual screen and the second virtual screen can be switched to the second perspective mentioned above.

[0074] As can be seen, the technical solutions provided by one or more embodiments of this disclosure can first capture scene images including a real screen using a first camera. Considering that the size of the real screen may be small, in order to improve the user's viewing experience, a virtual screen corresponding to the real screen can be generated from a first-view perspective. By mapping the video frame of the video stream to be played onto the virtual screen, a first video stream from the first-view perspective can be formed. This first video stream can then be fed back to the user's virtual reality device, allowing the user to view the first video stream from the first-view perspective. To provide a multi-view video viewing method, a second camera can also be deployed in the offline scene. Then, based on the relative position between the first camera and the second camera, the virtual screen from the first-view perspective can be converted into a virtual screen from the second-view perspective. Then, by mapping the video frame of the video stream to be played onto the virtual screen from the second-view perspective, a second video stream from the second-view perspective can be obtained, which can also be fed back to the user's virtual reality device. In this way, by providing users with video streams from different perspectives, users can choose a suitable viewing angle independently, thereby improving the user's video viewing experience.

[0075] Please see Figure 6 This disclosure also provides a video stream processing apparatus according to one embodiment, the apparatus comprising:

[0076] The first-view processing unit 100 is used to acquire scene images containing real screens captured by the first image acquisition device and generate virtual screens corresponding to the real screens in the first view.

[0077] The first mapping unit 200 is used to map the video frame of the video stream to be played onto a virtual screen under the first viewpoint, and generate a first video stream under the first viewpoint based on the first mapped frame on the virtual screen under the first viewpoint.

[0078] The viewpoint conversion unit 300 is used to convert the virtual screen under the first viewpoint into a virtual screen under the second viewpoint according to the relative position between the first image acquisition device and the second image acquisition device.

[0079] The second mapping unit 400 is used to map the video frame of the video stream to be played onto a virtual screen under the second perspective, and generate a second video stream under the second perspective based on the second mapped frame on the virtual screen under the second perspective.

[0080] The video stream feedback unit 500 is used to feed back the first video stream and the second video stream to the virtual reality device, so that the virtual reality device can play the first video stream or the second video stream from different perspectives.

[0081] The specific processing logic of each functional module can be found in the description of the aforementioned method implementation method, and will not be repeated here.

[0082] The various units described in the above embodiments can be implemented by a computer chip or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0083] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0084] Please see Figure 7 This disclosure also provides a server, which includes a memory and a processor. The memory is used to store a computer program, which, when executed by the processor, implements the above-described video stream processing method.

[0085] This disclosure also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described video stream processing method.

[0086] The processor can be a central processing unit (CPU). It can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0087] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described embodiments.

[0088] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0090] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus, devices, and storage media are basically similar to method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0091] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0092] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of processing a video stream, characterized by, The method comprises: acquiring a scene image containing a real screen photographed by a first image acquisition device, and generating a virtual screen corresponding to the real screen under a first view angle, comprising: generating a first virtual screen corresponding to the real screen at the position of the real screen; generating a second virtual screen corresponding to the real screen in the direction of the central axis of the shooting range of the second image acquisition device with the central axis as the reference; taking the first virtual screen and the second virtual screen as the virtual screen generated under the first view angle; mapping the video picture of the video stream to be played onto the virtual screen under the first view angle, and generating a first video stream under the first view angle according to the first mapping picture on the virtual screen under the first view angle; converting the virtual screen under the first view angle into a virtual screen under a second view angle according to the relative position between the first image acquisition device and the second image acquisition device; mapping the video picture of the video stream to be played onto the virtual screen under the second view angle, and generating a second video stream under the second view angle according to the second mapping picture on the virtual screen under the second view angle; feeding back the first video stream and the second video stream to a virtual reality device, so that the virtual reality device plays the first video stream or the second video stream under different view angles.

2. The method of claim 1, wherein, Generating a virtual screen corresponding to the real screen under the first view angle comprises: identifying at least one real corner point of the real screen in the scene image, and generating a virtual corner point corresponding to each of the real corner points; taking the area formed by a plurality of virtual corner points as a virtual screen under the first view angle.

3. The method of claim 2, wherein, Identifying a plurality of real corner points of the real screen in the scene image comprises: acquiring a calibration coefficient of the first image acquisition device, and converting the scene image into a corrected image through the calibration coefficient; cropping a region image containing the real screen from the corrected image, and identifying a plurality of real corner points of the real screen in the region image.

4. The method of claim 1, wherein, Mapping the video picture of the video stream to be played onto the virtual screen under the first view angle comprises: determining a plurality of groups of mapping points between the video picture and the virtual screen under the first view angle, and generating a mapping matrix based on the position information of the plurality of groups of mapping points; mapping each pixel point in the video picture of the video stream to be played onto the virtual screen under the first view angle through the mapping matrix.

5. The method according to claim 1 or 4, characterized in that, Generating a first video stream under the first view angle according to the first mapping picture on the virtual screen under the first view angle comprises: inverting projecting each of the first mapping pictures into a corresponding panoramic image according to the calibration coefficient of the first image acquisition device, and taking a video stream formed by each of the panoramic images as the first video stream under the first view angle.

6. The method of claim 1, wherein, Converting the virtual screen under the first view angle into a virtual screen under the second view angle comprises: mapping the virtual screen under the first view angle to a world coordinate system according to the external parameter of the first image acquisition device; According to the extrinsic parameter of the second image acquisition device, the virtual screen in the world coordinate system is mapped to a camera coordinate system of the second image acquisition device to form a virtual screen in the second view angle.

7. The method of claim 1, wherein, After the first video stream in the first view angle is generated, the method further includes: fusing the video picture of the first video stream with a scene image captured by the first image acquisition device to generate a first fused video stream, and feeding back the first fused video stream to a virtual reality device.

8. The method of claim 7, wherein, The fusing the video picture of the first video stream with the scene image captured by the first image acquisition device includes: for any color component in the video picture and the scene image, performing weighted summation on component data of the color component based on a preset fusion coefficient to generate fusion data corresponding to the color component; taking an image composed of the fusion data of each color component as a fused image.

9. The method of claim 1, wherein, After the second video stream in the second view angle is generated, the method further includes: fusing the video picture of the second video stream with a scene image captured by the second image acquisition device to generate a second fused video stream, and feeding back the second fused video stream to a virtual reality device.

10. The method of claim 1, wherein, The mapping the video picture of the to-be-broadcast video stream to the virtual screen in the first view angle includes: mapping the video picture of the to-be-broadcast video stream to the first virtual screen and the second virtual screen respectively; The converting the virtual screen in the first view angle to a virtual screen in a second view angle includes: converting the first virtual screen and the second virtual screen to the second view angle.

11. An apparatus for processing a video stream, the apparatus comprising: The apparatus includes: a first view angle processing unit configured to acquire a scene image containing a real screen captured by a first image acquisition device, and generate a virtual screen corresponding to the real screen in a first view angle, including: generating a first virtual screen corresponding to the real screen at a position where the real screen is located; generating a second virtual screen corresponding to the real screen in a direction of a central axis of a shooting range of the second image acquisition device with the central axis as a reference; and taking the first virtual screen and the second virtual screen as the virtual screen generated in the first view angle; a first mapping unit configured to map a video picture of a to-be-broadcast video stream to the virtual screen in the first view angle, and generate a first video stream in the first view angle according to a first mapping picture on the virtual screen in the first view angle; a view angle conversion unit configured to convert the virtual screen in the first view angle to a virtual screen in a second view angle according to a relative position between the first image acquisition device and a second image acquisition device; a second mapping unit configured to map the video picture of the to-be-broadcast video stream to the virtual screen in the second view angle, and generate a second video stream in the second view angle according to a second mapping picture on the virtual screen in the second view angle; a video stream feedback unit configured to feed back the first video stream and the second video stream to a virtual reality device, so that the virtual reality device plays the first video stream or the second video stream in different view angles.

12. A server, characterized by The server comprises a memory and a processor, the memory is used for storing a computer program, the computer program is executed by the processor to realize the method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium is used for storing a computer program, the computer program is executed by the processor to realize the method as claimed in any one of claims 1 to 10.

Images