Method and device for generating and playing image

[0041] In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

[0042] The steps shown in the flowcharts of the figures may be performed in a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0043] In the prior art, the panorama content is watched by the viewer adjusting the viewing angle of the panorama content through attitude sensors such as gravity sensors, acceleration sensors, and gyroscope sensors, or adjusting the viewing angle of the panorama content through gestures of the viewer. Because different viewers generally perform different playback control operations, that is to say, when different viewers watch the same panoramic content, they are likely to watch it from different angles of view. Due to the different viewing angles, the viewing experience will be different. And when a viewer shares a panorama content with others, the purpose is to enable the viewer to have similar feelings and resonate with himself. However, it is difficult for the current panorama playback method to enable the sharer and the sharee to obtain exactly the same viewing experience.

[0044] In this application, in order to enable different viewers to watch the panorama content with the same picture effect, while playing the panorama content, the pictures displayed on the screen are synthesized into the screen display content according to the sequence of picture display. For example, while playing the panoramic content, the rendering parameters of each panoramic image frame of the panoramic content can be recorded to obtain a rendering parameter sequence; the rendering parameter sequence can be saved, so that the panoramic content can be played as screen display content according to the rendering parameter sequence. Alternatively, the picture displayed on the screen can also be directly input to the encoder to be encoded as independent content as the screen display content. When a user plays or shares panoramic content to other people, he can enable others to have the same viewing experience as himself by playing or sharing the on-screen content.

[0045] The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0046] like figure 1 As shown, the method for generating an image of the present application includes:

[0047] Step 101: Obtain panoramic content;

[0048] Step 102: Play the panoramic content;

[0049] Step 103: While playing the panorama content, synthesize the pictures displayed on the screen into screen display content according to the order in which the pictures are displayed.

[0050] The method for generating images provided in this application can be applied to display devices such as mobile phones and head-mounted display devices for playing panoramic content, and can also be applied to scenes where panoramic content is post-processed. Panoramic content can be panoramic pictures, panoramic videos, etc. taken by shooting equipment such as panoramic cameras and sports cameras, or panoramic movies, panoramic animations, etc., which are shot and produced by professional equipment.

[0051] First of all, in order to implement the method for generating images in this application, it is necessary to obtain the panorama content first. For example, for panoramic videos and panoramic pictures shot by shooting equipment, the panoramic content can be obtained directly from the shooting device; for panoramic animation and computer-generated panoramic content, the panoramic content can be obtained from the corresponding production equipment.

[0052] Secondly, in order to obtain the on-screen content, after obtaining the panoramic content, it is necessary to perform panoramic playback on the panoramic content.

[0053]The panorama content is a 360-degree spatial image, and the viewing direction intended to be displayed in the panorama content is a display angle of view. When the panoramic content is displayed in the display area, the picture within the window range is intercepted in the direction of the corresponding display viewing angle of the panoramic content, mapped into a two-dimensional image according to a certain projection transformation, and the two-dimensional image is correspondingly displayed in the display area. The window range is usually a rectangular area. The picture within the window range is displayed on the screen. The projection transformation mentioned above includes but not limited to perspective transformation, orthographic projection transformation and the like.

[0054] When displaying panoramic content, viewers can perform specific input operations on the display area, such as touch and slide operations on the touch screen, so as to change the viewing angle direction of the display area, zoom the display area, etc., and also dynamically change the size of the display area and shape etc.

[0055] Thirdly, while playing the panorama content, the pictures displayed on the screen are synthesized into the screen display content according to the order in which the pictures are displayed.

[0056] In this application, no matter when the panoramic content is played on the playback device for viewers to enjoy, or when the panoramic content is edited and produced later, the pictures in the panoramic content displayed on the screen can be synthesized into one screen display content , so that other viewers in the future can obtain the same visual experience as the current viewer by watching the on-screen content. This method can prevent different viewers from viewing the same panoramic content from different perspectives and cannot produce a similar visual experience, and achieve the purpose of sharing the panoramic content and resonating with other viewers for the current viewer.

[0057] In an optional embodiment of the present application, the above-mentioned screens displayed on the screen are synthesized into screen display content according to the order in which the screens are displayed, including:

[0058] Record the rendering parameters of each panoramic image frame when playing the panoramic content, and obtain the rendering parameter sequence;

[0059] The rendering parameter sequence is saved, so that the panorama content can be played as screen display content according to the rendering parameter sequence.

[0060] After the rendering parameter sequence is obtained, when the panoramic content is played again, the rendering parameter sequence is used to render the panoramic content, so that the same viewing effect as that of the current viewer can be obtained.

[0061] Further, the rendering parameters include: viewing position information, window description information and frame information.

[0062] The viewfinder position information is used to determine the position of viewfinder in the unit circle of the panorama content. When playing the on-screen content, it is necessary to first determine the viewing position in the unit circle of the panoramic content according to the viewing position information, so as to perform viewing from the viewing position in the unit circle of the panoramic content.

[0063] The window description information is used to determine the position and range of the window in the unit circle. The view window is used to capture a picture from the panoramic content in the direction of the display viewing angle, and the captured picture is then mapped into a two-dimensional image according to a certain projection transformation and displayed in the display area. When playing the on-screen display content, after determining the viewfinder position, determine the position and range of the window in the unit circle according to the window description information, then take a view from the viewfinder position, determine the picture within the window range, and render it to the window for display By.

[0064] The frame information is used to establish a corresponding relationship between rendering parameters and panoramic image frames in the panoramic content. The rendering parameters of multiple panoramic image frames are included in the rendering parameter sequence, so the corresponding relationship between each panoramic image frame of the panoramic content and its respective rendering parameters needs to be established through frame information. When playing the on-screen content, it is first necessary to determine the rendering parameters corresponding to the panoramic image frame according to the frame information. The frame information can be the playing time of the image frame, or the frame ID, etc.

[0065] Further, the window description information includes the coordinates of the intersection point of the line of sight and the unit circle, the horizontal expansion angle of the window, and the vertical expansion angle of the window. In order to determine the frame captured by the window in the panoramic content, it is necessary to determine the position and range of the window in the unit circle of the panoramic content. A line of sight is a straight line indicating the direction of the viewing angle. The intersection of the line of sight and the unit circle of the panoramic content is the center of the window, and the position of the window on the unit circle can be determined by the coordinates of the intersection of the line of sight and the unit circle. The horizontal expansion angle of the window is used to determine the horizontal expansion range of the window. Similarly, the vertical expansion angle of the window is used to determine the vertical expansion range of the window. The range delineated by the window on the unit circle is jointly determined by the included angle of horizontal expansion and the included angle of vertical expansion of the window.

[0066] like figure 2 As shown, assuming that at time t, the viewfinder position information is (x c ,y c ,z c ), the coordinates of the intersection of the line of sight and the unit circle in the window description information are (x e ,y e ,z e ), the horizontal expansion angle of the window is FovX, and the vertical expansion angle of the window is FovY. The center of the unit circle is (x 0 ,y 0 ,z 0 ), with the center at (x c ,y c ,z c ) and (x e ,y e ,z e ) connection or an extension of the connection. The rendering parameters corresponding to the image frame at time t can be expressed as: (t,x c ,y c ,z c ,x e ,y e ,z e , FovX, FovY).

[0067] By analogy, the rendering parameter sequence composed of the rendering parameters of each panoramic image frame can be expressed as:

[0068] {

[0069] (t 1 ,x c1 ,y c1 ,z c1 ,x e1 ,y e1 ,z e1 , FovX1, FovY1)

[0070] (t 2 ,x c2 ,y c2 ,z c2 ,x e2 ,y e2 ,z e2 , FovX2, FovY2)

[0071] …

[0072] (t n ,x cn ,y cn ,z cn ,x en ,y en ,z en , FovXn, FovYn)

[0073]}

[0074] All the rendering parameters generated by one playback of the panoramic content are recorded, so that they can be used to regenerate the picture sequence generated by this playback. When playing again, as long as the panoramic image frame of the decoded panoramic content is rendered according to the rendering parameters recorded in this sequence, the image effect of the last playback can be restored.

[0075] The method for generating images provided by this application is mainly used to generate on-screen content for panoramic videos. Panoramic videos can be regarded as a sequence of panoramic pictures and contain many pictures. Therefore, when playing panoramic videos, it is more important to different viewers. It is easy to produce different visual experiences. When generating screen display content for a panoramic video, one panoramic image frame of the panoramic video may correspond to one rendering parameter, or may correspond to multiple rendering parameters. For example, if a panoramic image frame of a panoramic video is viewed in multiple viewing directions, that is to say, multiple pictures in the panoramic image frame are displayed on the screen, then when generating screen display content, the panoramic image frame It is necessary to correspond to a plurality of rendering parameters, so that the on-screen display content can include multiple images of the panoramic image frame displayed on the screen.

[0076] The image generation method provided in this application can also be used to generate screen display content for panoramic pictures. Similar to the panoramic image frame contained in the panoramic video, although the panoramic picture is only one frame of image, it contains a 360-degree picture. screen, record multiple rendering parameters for the panorama picture, and also obtain the screen display content of the panorama picture.

[0077] In another optional embodiment of the present application, the actual screen display can also be recorded and exported as an ordinary video. When this video is played again, no special processing is required to reproduce the current viewer obtained when playing the panoramic content. screen shot. Such a video can be directly shared with others, and as long as it is played normally, the current viewer can see the picture effect when recording and exporting without panoramic video processing.

[0078] When recording and exporting the picture actually displayed on the screen into an ordinary video, the picture displayed on the screen can be input to the encoder for encoding, and the encoder encodes the input picture into independent content, and the independent content obtained by encoding as screen content.

[0079] This method is also applicable to panoramic videos and panoramic pictures.

[0080] like image 3 As shown, the application also provides a method for playing images, including:

[0081] Step 301: Obtain the panoramic content and the rendering parameter sequence corresponding to the panoramic content. The rendering parameter sequence includes the rendering parameters corresponding to each frame of the panoramic image in the panoramic content. The rendering parameters are the panoramic images recorded during the last panoramic playback of the panoramic content. The parameters used for rendering;

[0082] Step 302: Render and display the panoramic image frame according to the rendering parameters corresponding to each panoramic image frame of the panoramic content, so that the panoramic content is played according to the same screen display content as the last panoramic playback.

[0083] In order to display the panorama content with the same picture effect as the previous playback, while acquiring the panorama content, it is necessary to acquire the corresponding rendering parameter sequence of the panorama content.

[0084] The rendering parameter sequence contains the rendering parameters corresponding to each panoramic image frame of the panoramic content. Rendering the panoramic image frame according to its corresponding rendering parameters in the rendering parameter sequence can restore the panoramic image frame when the panoramic content was played last time. screen effect. For the entire panoramic content, rendering each panoramic image frame according to the corresponding rendering parameters can restore the overall picture effect of the last playback of the panoramic content, so that different viewers can obtain the same view for the same panoramic content. viewing experience.

[0085] Further, the rendering parameters include: viewing position information, window description information and frame information.

[0086] The above-mentioned rendering and display of the panoramic image frame according to the corresponding rendering parameters of each panoramic image frame of the panoramic content includes:

[0087] Determine the rendering parameters corresponding to the panoramic image frame according to the frame information in the rendering parameters;

[0088] Determine the viewfinder position in the unit circle of the panorama content according to the viewfinder position information in the rendering parameters;

[0089] Determine the position and range of the window in the unit circle according to the window description information in the rendering parameters;

[0090] Framing is performed from the viewfinder position, and the picture enclosed in the view window is rendered into the view window and displayed to the viewer.

[0091]When playing the panoramic content according to the same screen content as the last time, it is necessary to first determine the corresponding rendering parameters for the panoramic image frame in the rendering parameter sequence according to the frame information in the rendering parameters; for example, according to the frame ID or time information Match the rendering parameters corresponding to the panoramic image frame in the rendering parameter sequence.

[0092] For each panoramic image frame of the panoramic content, after matching the corresponding rendering parameters, determine the viewing position in the unit circle of the panoramic content according to the viewing position information in the rendering parameters; then, according to the window description information in the rendering parameters in the unit The position and range of the window are determined on the circle; finally, the viewfinder will be framed from the viewfinder position, and the picture enclosed in the window will be rendered to the window for display to the viewer.

[0093] Specifically, when rendering the panoramic image frame at time t during playback, set the viewing position at (xc, yc, zc) according to the viewing position information, and determine the center position of the window in the unit circle as (xe, ye) according to the window description information , ze), and according to the window horizontal expansion angle FovX and the window vertical expansion angle FovY in the window description information, determine the scope of the window delineation, and finally, view the view from the viewfinder position, delineate the 360-degree panoramic image frame within the window range The picture is rendered to the window and displayed on the screen, that is, the picture displayed on the screen at time t when the panorama content is played last time is reproduced.

[0094] In this application, the sequence of images displayed when the panoramic content was played last time can be regenerated by using the rendering parameter sequence recorded when the panoramic content was played last time. When playing again, it is only necessary to render each panoramic image frame of the panoramic content according to its corresponding rendering parameters, and the picture effect of the last panoramic playback of the panoramic content can be restored.

[0095] In another optional embodiment of the present application, when the screen display recording of the panoramic content is exported as a common video, the screen picture obtained when the user plays the panoramic content can be reproduced without special processing.

[0096] like Figure 4 As shown, the present application also provides a device for generating an image, including:

[0097] an acquisition unit, used to acquire panoramic content;

[0098] a playback unit, used to play the panoramic content;

[0099] An on-screen content synthesizing unit is used for synthesizing the pictures displayed on the screen into the on-screen content according to the sequence of picture display while the playback unit is playing the panorama content.

[0100] Further, the on-screen content synthesizing unit synthesizes the pictures displayed on the screen into the on-screen content according to the order in which the pictures are displayed, including:

[0101] Record the rendering parameters of each panoramic image frame when playing the panoramic content, and obtain the rendering parameter sequence;

[0102] The rendering parameter sequence is saved, so that the panorama content can be played as screen display content according to the rendering parameter sequence.

[0103] Further, the on-screen content synthesizing unit synthesizes the pictures displayed on the screen into the on-screen content according to the order in which the pictures are displayed, including:

[0104] The picture displayed on the screen is input to the encoder to be encoded as independent content, and the encoded independent content is used as the screen display content.

[0105] Further, the rendering parameters include: viewfinder position information, window description information and frame information;

[0106] The viewfinder position information is used to determine the position of viewfinder in the unit circle of the panorama content;

[0107] The window description information is used to determine the position and range of the window in the unit circle;

[0108] The frame information is used to establish a corresponding relationship between rendering parameters and panoramic image frames in the panoramic content.

[0109] Further, the window description information includes the coordinates of the intersection point of the line of sight and the unit circle, the horizontal expansion angle of the window, and the vertical expansion angle of the window.

[0110] like Figure 5 As shown, the present application also provides a device for playing images, including:

[0111] An acquisition unit, used to acquire the panoramic content and the rendering parameter sequence corresponding to the panoramic content, the rendering parameter sequence includes the rendering parameters corresponding to each panoramic image frame in the panoramic content, and the rendering parameters are recorded when the panoramic content is played last time The parameters used to render the panoramic image frame;

[0112] A rendering unit, configured to render and display the panoramic image frame according to the corresponding rendering parameters of each panoramic image frame of the panoramic content, so as to play the panoramic content according to the same screen display content as the last panoramic playback.

[0113] Further, the rendering unit renders and displays the panoramic image frame according to the rendering parameters corresponding to each panoramic image frame of the panoramic content, including:

[0114] Determine the rendering parameters corresponding to the panoramic image frame according to the frame information in the rendering parameters;

[0115] Determine the viewfinder position in the unit circle of the panorama content according to the viewfinder position information in the rendering parameters;

[0116] Determine the position and range of the window in the unit circle according to the window description information in the rendering parameters;

[0117] The viewfinder will be framed from the viewfinder position, and the picture framed in the window will be rendered into the window and displayed to the viewer.

[0118] Further, the present application also provides another device for generating images, including:

[0119] processor; and

[0120] memory for storing processor-executable instructions;

[0121] where the processor is configured as:

[0122] Get panorama content;

[0123] Play the panoramic content;

[0124] While playing the panorama content, the pictures displayed on the screen are synthesized into the screen display content according to the order in which the pictures are displayed.

[0125] Further, the present application also provides another device for playing images, including:

[0126] processor; and

[0127] memory for storing processor-executable instructions;

[0128] where the processor is configured as:

[0129] Get the panoramic content and the corresponding rendering parameter sequence of the panoramic content. The rendering parameter sequence contains the rendering parameters corresponding to each panoramic image frame in the panoramic content. The rendering parameters are the rendering of the panoramic image frame recorded when the panoramic content was played last time. the parameters used;

[0130] The panoramic image frames are rendered and displayed according to the corresponding rendering parameters of each panoramic image frame of the panoramic content, so that the panoramic content is played according to the same screen display content as the last panoramic playback.

[0131] Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

[0132] The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce in the process of realization Figure 1 process or processes and/or blocks Figure 1 means for the function specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The device is implemented in the process Figure 1 process or processes and/or blocks Figure 1 the function specified in a box or boxes.

[0134] These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that Instructions are provided for implementing the process in Figure 1 process or processes and/or blocks Figure 1 steps of the function specified in the box or boxes.

[0135] In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

[0136] Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

[0137] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

[0138] It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

[0139]Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

[0140] The above descriptions are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.