Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: determining each first video picture displayed through a visual area of a panoramic video and first image data of each first video picture; wherein in the playing process of the panoramic video, each first video picture is displayed in a visual area through the coordinate transformation matrix, and when the panoramic video is rotated to different angles, the first video pictures in the visual area are different; drawing a second video picture in the pixel buffer area based on first image data corresponding to the first video picture; and compressing the second video image in the pixel buffer area to obtain the first thumbnails, and displaying each first thumbnail on the preview interface, thereby ensuring that the first video image is correspondingly consistent with the first thumbnail, enabling a user to check the video content of each first video image based on the first thumbnail on the preview interface, and improving the user experience. And the watching experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of digital media technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium. Background Art

[0002] Panoramic video is developed and extended on the basis of 360-degree panoramic technology, which converts static panoramic pictures into dynamic video images, giving users a sense of immersion. During the playback of panoramic video, in order to facilitate users to view the video content included in the panoramic video, a thumbnail corresponding to the panoramic video will be generated on the playback interface of the panoramic video.

[0003] In the related art, although directly compressing the decoded picture can obtain the thumbnail corresponding to the panoramic video, this method is not conducive to users accurately positioning the target nodes in the thumbnail and is not conducive to improving the user's viewing experience. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image processing method and apparatus, an electronic device, and a storage medium. To overcome the problems that users cannot accurately position the target nodes in the thumbnail and are not conducive to improving the user's viewing experience.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Determine each first video frame displayed through the visual area of the panoramic video and first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visual area through a coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames in the visual area are different;

[0007] Draw a second video frame in a pixel buffer based on the first image data corresponding to the first video frame;

[0008] Perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display each of the first thumbnails on a preview interface.

[0009] In some embodiments, the determining each first video frame displayed through the visual area of the panoramic video and first image data of each of the first video frames includes:

[0010] During the execution of a first adjustment operation on the panoramic video, determine each first video frame displayed through the visual area of the panoramic video and first image data of each of the first video frames; or

[0011] When the execution of the first adjustment operation ends, determine each first video frame displayed through the visible area of the panoramic video and the first image data of each of the first video frames.

[0012] In some embodiments, the method further includes:

[0013] When a trigger operation on a target node on the preview interface is detected, determine the first thumbnail located at the target node;

[0014] Display the first video frame corresponding to the first thumbnail located at the target node.

[0015] In some embodiments, the method further includes:

[0016] Detect a second adjustment operation;

[0017] Determine each third video frame displayed through the visible area of the panoramic video and the second image data of each of the third video frames; wherein, during the playback of the panoramic video, each of the third video frames is displayed in the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, and as the panoramic video rotates to different angles, the third video frames in the visible area are different;

[0018] Draw a fourth video frame in the pixel buffer based on the second image data corresponding to the third video frame;

[0019] Perform compression processing on the fourth video frame in the pixel buffer to obtain a second thumbnail;

[0020] Update the content displayed on the preview interface based on the second thumbnail.

[0021] In some embodiments, the method further includes:

[0022] If no operation on the panoramic video is detected within a preset duration, determine that the execution of the first adjustment operation ends.

[0023] In some embodiments, the method further includes:

[0024] Store each of the first thumbnails obtained by compressing each of the second video frames into the pixel buffer;

[0025] The displaying of the first thumbnail on the preview interface includes:

[0026] Select, from each of the first thumbnails, the first thumbnail at the target moment indicated by the application operation, and display the first thumbnail at the target moment on the preview interface.

[0027] According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0028] A first determination module, configured to determine each first video frame displayed through a visible area of a panoramic video and first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, and when the panoramic video rotates to different angles, the first video frames in the visible area are different;

[0029] A first drawing module, configured to draw a second video frame in a pixel buffer based on the first image data corresponding to the first video frame;

[0030] A first execution module, configured to perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display each of the first thumbnails on a preview interface.

[0031] In some embodiments, the first determination module is specifically configured to:

[0032] During the execution of a first adjustment operation on the panoramic video, determine each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames; or

[0033] After the execution of the first adjustment operation ends, determine each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames.

[0034] In some embodiments, the image processing apparatus further includes:

[0035] A second determination module, configured to determine the first thumbnail located at a target node when a trigger operation on the target node on the preview interface is detected;

[0036] A first display module, configured to display the first video frame corresponding to the first thumbnail located at the target node.

[0037] In some embodiments, the image processing apparatus further includes:

[0038] A first detection module, configured to detect a second adjustment operation;

[0039] A third determination module, configured to determine each third video frame displayed through the visible area of the panoramic video and second image data of each of the third video frames; wherein, during the playback of the panoramic video, each of the third video frames is displayed within the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, and as the panoramic video rotates to different angles, the third video frames within the visible area are different;

[0040] A second drawing module, configured to draw a fourth video frame in a pixel buffer based on the second image data corresponding to the third video frame;

[0041] A second execution module, configured to perform compression processing on the fourth video frame in the pixel buffer to obtain a second thumbnail;

[0042] An update module, configured to update the content displayed on the preview interface based on the second thumbnail.

[0043] In some embodiments, the image processing device further includes:

[0044] A fourth determination module, configured to determine that the execution of the first adjustment operation ends if no operation on the panoramic video is detected within a preset duration.

[0045] In some embodiments, the image processing device further includes:

[0046] A storage module, configured to store each of the first thumbnails obtained by compressing each of the second video frames into the pixel buffer;

[0047] The first execution module is specifically configured to:

[0048] Select, from each of the first thumbnails, the first thumbnail at the target moment indicated by the application operation, and display the first thumbnail at the target moment on the preview interface.

[0049] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0050] A processor;

[0051] A memory for storing processor-executable instructions;

[0052] Wherein, the processor is configured to: when executing the executable command, implement the steps in any one of the image processing methods in the first aspect above.

[0053] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, including:

[0054] When the instructions in the storage medium are executed by a processor of an electronic device, the steps in any one of the above-mentioned first aspect of the image processing method are implemented.

[0055] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0056] In the embodiments of the present disclosure, first, each first video frame displayed in the visible area of the panoramic video and the first image data of each of the first video frames are determined; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames in the visible area are different; then, a second video frame is drawn in a pixel buffer based on the first image data corresponding to the first video frame; finally, the second video frame in the pixel buffer is compressed to obtain a first thumbnail, and the first thumbnail is displayed on a preview interface.

[0057] In this way, compared with the prior art in which the frame after directly decoding the panoramic video is compressed to generate a thumbnail, in the embodiments of the present disclosure, by using a coordinate transformation matrix during the playback of the panoramic video, the first image data corresponding to each first video frame displayed in the visible area of the panoramic video is determined as the image data for generating the first thumbnail, so that the first video frame located in the visible area is consistent with the first thumbnail, Figure 1 and enables the user to view the video content of each first video frame based on the first thumbnail on the preview interface, thereby being beneficial to improving the user's viewing experience.

[0058] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0060] Figure 1 is a flowchart showing an image processing method according to an exemplary embodiment;

[0061] Figure 2 is a display diagram showing a display interface of an electronic device according to an exemplary embodiment;

[0062] Figure 3 is a generation diagram showing a thumbnail corresponding to a panoramic video in a related art according to an exemplary embodiment;

[0063] Figure 4It is a schematic diagram of the generation of a panoramic video shown according to an exemplary embodiment;

[0064] Figure 5 It is a schematic diagram of the generation of a thumbnail corresponding to a panoramic video shown according to an exemplary embodiment;

[0065] Figure 6 It is a block diagram of an image processing device shown according to an exemplary embodiment;

[0066] Figure 7 It is a hardware structure block of an electronic device shown according to an exemplary embodiment Figure 1 ;

[0067] Figure 8 It is a hardware structure block of an electronic device shown according to an exemplary embodiment Figure 2 . Detailed implementation manners

[0068] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0069] Figure 1 It is a flowchart of an image processing method shown according to an exemplary embodiment Figure 1 , as Figure 1 shown, the image processing method mainly includes the following steps:

[0070] In step 101, determine each first video frame displayed in the visible area of the panoramic video and the first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames in the visible area are different;

[0071] In step 102, draw a second video frame in the pixel buffer based on the first image data corresponding to the first video frame;

[0072] In step 103, perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display the first thumbnail on the preview interface.

[0073] It should be noted that the image processing method proposed in this disclosure can be applied to an electronic device. Here, the electronic device may include: a terminal device, for example, a mobile terminal or a fixed terminal. Among them, the mobile terminal may include: devices such as a mobile phone, a tablet computer, a laptop computer, a wearable electronic device, etc. The fixed terminal may include: a desktop computer, a smart TV, a vehicle-mounted device, etc. In some other embodiments, the image processing method can also be applied to an application installed on the electronic device.

[0074] In some other embodiments, the image processing method in the embodiments of this disclosure can be configured in an image processing device, and this image processing device can be set in the electronic device, and the embodiments of this disclosure do not limit this. It should be noted that the execution subject of the embodiments of this disclosure can be the central processing unit (CPU) in the electronic device in terms of hardware, and can be the relevant background service in the electronic device in terms of software, and this is not limited.

[0075] Exemplarily, Figure 2 is a schematic diagram of a display interface of an electronic device shown according to an exemplary embodiment, as Figure 2 shown, A1 is a playback interface, A2 is a progress bar corresponding to the panoramic video, and the progress bar is a thumbnail of the panoramic video at multiple time points. By observing the multiple thumbnails on the progress bar, the user can understand the video content of the panoramic video.

[0076] Figure 3 is a schematic diagram of the generation of a thumbnail corresponding to a panoramic video in a related technology shown according to an exemplary embodiment, as Figure 3 shown, the thumbnails B2 in the related technology are all obtained by directly compressing the decoded picture B1, so as to obtain the thumbnail corresponding to the panoramic video. During the playback of the panoramic video, if the user wants to adjust the viewing angle of the panoramic video, the thumbnail will not change. Therefore, there will be a problem that the thumbnail is inconsistent with the playback picture, making it difficult for the user to accurately locate the thumbnail corresponding to the target node through the progress bar, which is not conducive to improving the user's operation experience.

[0077] Therefore, the embodiments of this disclosure first determine each first video picture displayed through the visible area of the panoramic video and the first image data of each of the first video pictures; then generate a first thumbnail based on each first image data, so that the picture content of the generated first thumbnail is consistent with the picture content of the first video picture, thereby being conducive to improving the user's viewing experience.

[0078] Here, the panoramic video can be played through an electronic device, such as a computer, a mobile phone or a dedicated head-mounted display, and the embodiments of this disclosure do not limit this.

[0079] It can be understood that panoramic video can be referred to as surround video or immersive video, which is developed and extended based on the technology of 360-degree or 720-degree panorama, converting static panoramic pictures into dynamic video images. Therefore, during the playback of panoramic video, limited by the viewing angle range of the human eye and the viewing angle range of the playback device, users can only see a part of the panoramic area in the panoramic video.

[0080] It should be noted that the decoded panoramic video frame will not be directly rendered onto the playback interface. Instead, it is first rendered onto a model through a 3D rendering tool (such as the open graphics library OpenGL) to form a panoramic video. At the same time, the camera (i.e., the playback interface of the terminal device for playing panoramic video) is placed inside the panoramic video, so that the video frame observed by the user is the panoramic frame inside the model.

[0081] Here, the panoramic video can be a spherical panoramic video or a circular panoramic video, and the embodiments of the present disclosure do not limit this.

[0082] Here, the Open Graphics Library (OpenGL) is a cross-language and cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. Using OpenGL, triangles can be drawn to construct geometric figures, thereby constructing a panoramic video in a virtual environment.

[0083] Exemplarily, Figure 4 is a schematic diagram of the generation of a panoramic video shown according to an exemplary embodiment, as Figure 4 shown, C1 is the panoramic video frame, C2 is the spherical panoramic video, and the formation process of the spherical panoramic video can be as follows:

[0084] Using the initialized OpenGL, triangles are drawn to construct geometric figures, and the number of sides of the polyhedron is continuously increased, thereby constructing a sphere model in a virtual environment. Then, the coordinates of each vertex on the sphere model and the coordinates of each pixel point in the decoded panoramic video frame are determined. Next, the coordinates of each pixel point are corresponding to the coordinates of each vertex, realizing the rendering of the decoded panoramic video frame onto the surface of the sphere model, thereby forming the spherical panoramic video.

[0085] It can be understood that only one first video frame can be displayed in the visible area of the panoramic video. Therefore, when the panoramic video rotates to different angles, the first video frame displayed in the visible area is different.

[0086] It should be explained that after determining the first image data corresponding to each first video frame displayed through the visible area of the panoramic video, a second video frame can be drawn in the pixel buffer based on the first image data.

[0087] Here, the pixel buffer is an off-screen pixel buffer that can store pixel data, texture data, vertex data, etc., and the embodiments of the present disclosure do not limit this. At the same time, the size of the pixel buffer can be set arbitrarily, and the embodiments of the present disclosure do not limit this.

[0088] Exemplarily, the pixel buffer can be a Pixel Buffer Object (PBO) for storing pixel data. Here, the method of establishing the pixel buffer object is usually implemented based on the internal method of OpenGL, which will not be elaborated here.

[0089] In some embodiments, in order to ensure the rendering effect of the second video frame, after obtaining the first image data of the first video frame within the visible area of the panoramic video, the first image data can be preprocessed, and each second video image can be rendered in real time in the pixel buffer. Among them, the preprocessing can be a patterned processing of the video frame, such as modifying the color mode of the video frame, the size of the video frame, or performing filtering, distortion correction, etc.

[0090] It can be understood that after each second video frame is rendered, the second video frames in the pixel buffer can be compressed to obtain a first thumbnail, so as to facilitate the user to obtain the video content within the visible area.

[0091] Here, the first thumbnail refers to a small image obtained after the image in the computer is processed by a compression method, and has the function of previewing the content of the panoramic video frame.

[0092] The image format of the first thumbnail can be the png format, the jpg format, or the bmp format, and the embodiments of the present disclosure do not limit this.

[0093] Here, the format of the rendered second video image can be the RGB format, and a format processing tool, such as a multimedia video processing tool, can be used to convert the RGB format into the png format or the bmp format to obtain the first thumbnail.

[0094] Exemplarily, the first image data corresponding to each first video frame in the visible area of the panoramic video is rendered into the off-screen pixel buffer through OpenGL rendering functions (such as glDrawArrays and eglSwapBuffers) to obtain each second video frame, and then the RGB data corresponding to the second video frame in the off-screen pixel buffer is read into the memory through an OpenGL reading function (such as glReadPixels). Finally, through a format processing tool (such as ffmpeg), the RGB data is converted into png data or bmp data to obtain the first thumbnail.

[0095] It should be noted that, in order to improve the user's viewing experience, after generating the first thumbnail corresponding to each first video frame, the first thumbnail can be displayed on the preview interface so that the user can pre-understand the video content of the panoramic video based on the respective thumbnails during the playback of the panoramic video.

[0096] Here, the complete first thumbnail can be displayed on the preview interface, or a partial first thumbnail can be displayed on the preview interface. The embodiments of the present disclosure do not limit this.

[0097] Meanwhile, the display position of the first thumbnail can also be set arbitrarily. The embodiments of the present disclosure do not limit this.

[0098] In the embodiments of the present disclosure, for the playback scenario of the panoramic video and the generation scenario of the first thumbnail, the decoded panoramic video frames need to be rendered onto a model to form a panoramic video, and then based on the coordinate transformation matrix, each first video frame within the visible area of the panoramic video is determined.

[0099] Exemplarily, for the playback scenario of the panoramic video, based on the first coordinate transformation matrix, each first video frame within the visible area of the panoramic video is determined. The image data of each first video frame can be rendered onto the screen through view transformation or mapping transformation, so that each first video frame is displayed on the playback interface of the screen.

[0100] Another exemplarily, for the generation scenario of the first thumbnail, based on the second coordinate transformation matrix, the first image data corresponding to each first video frame within the visible area of the panoramic video is determined. Then, based on the first image data, a second video frame is drawn in the pixel buffer, and the second video frame is compressed to generate the thumbnail corresponding to each first video frame.

[0101] Here, the first coordinate transformation matrix and the second coordinate transformation matrix are consistent, that is, the video frames within the visible area of the panoramic video determined using the first coordinate transformation matrix are the same as those determined using the second coordinate transformation matrix.

[0102] In this way, the first video frame drawn onto the screen is consistent with the second video frame drawn into the pixel buffer, and the first thumbnail is obtained by compressing the second video frame in the pixel buffer. Therefore, during the playback of the panoramic video, the content of the frame displayed on the playback interface is consistent with the content of the first thumbnail displayed on the preview interface.

[0103] In some embodiments, the automatic update time of the first thumbnail can be set. For example, if the automatic update time is set to 5 seconds, every 5 seconds, the first image data corresponding to each first video frame within the visible area of the panoramic video will be automatically determined, and a second video frame will be drawn in the pixel buffer based on the first image data. Finally, the second video frame will be compressed to obtain the first thumbnail.

[0104] In the embodiments of the present disclosure, first, the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames are determined. Among them, during the playback of the panoramic video, each of the first video frames is displayed within the visible area through a coordinate transformation matrix. As the panoramic video rotates to different angles, the first video frames within the visible area are different. Then, a second video frame is drawn in the pixel buffer based on the first image data corresponding to the first video frame. Finally, the second video frame in the pixel buffer is compressed to obtain the first thumbnail, and the first thumbnail is displayed on the preview interface.

[0105] Thus, compared with the prior art in which the frame after decoding the panoramic video is directly compressed to generate a thumbnail, in the embodiments of the present disclosure, the first image data corresponding to each first video frame displayed in the visible area of the panoramic video obtained through the coordinate transformation matrix during the playback of the panoramic video is determined as the image data for generating the first thumbnail, so that the first video frame located within the visible area is consistent with the first thumbnail, Figure 1 and enables the user to view the video content of each first video frame based on the first thumbnail on the preview interface, thereby being beneficial to improving the user's viewing experience.

[0106] In some embodiments, the determination of the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames includes:

[0107] During the execution of the first adjustment operation on the panoramic video, determine the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames; or

[0108] In the case where the first adjustment operation is completed, determine the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames.

[0109] It should be noted that during the playback of the panoramic video, the user will output a first adjustment operation on the panoramic video, changing the viewing angle of the panoramic video, so as to be able to view different positions in the panoramic area of the panoramic video.

[0110] It can be understood that during the execution of the first adjustment operation, the panoramic video rotates to different angles, and the first video frame displayed in the visible area of the panoramic video keeps changing.

[0111] Exemplarily, if the first adjustment operation applied to the panoramic video is a sliding operation, during the execution of the sliding operation, the coordinate transformation matrix inside the panoramic video will be correspondingly changed to form visible areas at different angles.

[0112] In some embodiments, if the sliding operation is from left to right on the playback interface, in response to the sliding operation, the panoramic video will rotate to the right by a certain angle. As the coordinate transformation matrix changes, visible areas at different angles will be formed. Then, the first image data within the visible areas at different angles is acquired to generate a first thumbnail corresponding to the sliding operation.

[0113] In other embodiments, if the sliding operation is from right to left on the playback interface, in response to the sliding operation, the panoramic video will rotate to the left by a certain angle. As the coordinate transformation matrix changes, visible areas at different angles will be formed. Then, the first image data within the visible areas at different angles is acquired to generate a first thumbnail corresponding to the sliding operation.

[0114] It can be understood that to ensure the generation efficiency of the first thumbnail, during the execution of the first adjustment operation applied to the panoramic video, it can be determined in real time the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames, so as to facilitate the real-time drawing of the second video frame in the pixel buffer to obtain the first thumbnail.

[0115] And to ensure the generation effect of the first thumbnail, it can be determined only when the first adjustment operation is completed the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames, so as to ensure the integrity of the first image data, and to facilitate the drawing of the second video frame consistent with the first video frame in the pixel buffer.

[0116] In the embodiments of the present disclosure, during the execution of the first adjustment operation on the panoramic video, the first video frames and the first image data of each of the first video frames displayed through the visible area of the panoramic video can be determined in real time, so that the second video frame can be drawn in the pixel buffer based on the first image data obtained in real time, thereby improving the efficiency of generating the first thumbnail; alternatively, when the first adjustment operation is completed, the first video frames displayed through the visible area of the panoramic video and the first image data of each of the first video frames can be determined, so as to ensure the integrity of the first image data and improve the generation effect of the first thumbnail.

[0117] In some embodiments, the method further includes:

[0118] When a trigger operation on the target node on the preview interface is detected, determine the first thumbnail located at the target node;

[0119] Display the first video frame corresponding to the first thumbnail located at the target node.

[0120] It can be understood that when the user is watching the panoramic video, the user can move forward or backward the video content. For example, the user can drag the progress bar to adjust the playback speed of the video, so as to fast forward or rewind the video to a certain time point for playback. Therefore, in order to further improve the user's viewing experience, a progress bar corresponding to each second video frame can be generated for the first thumbnail, so that the user can adjust each node in the progress bar to locate the first video frame that meets their own needs.

[0121] It should be explained that the first thumbnails corresponding to different nodes are different. In order to improve the accuracy of jumping to the video frame that the user is interested in, in the embodiments of the present disclosure, when a trigger operation on the target node on the preview interface is detected, first determine the first thumbnail located at the target node, then display the first video frame corresponding to the first thumbnail of the target node on the playback interface, and then start playing the panoramic video with the first video frame as the playback starting point.

[0122] Here, the shape and size of the target node can be set arbitrarily. At the same time, the display position of the progress bar can also be set arbitrarily, and the embodiments of the present disclosure do not limit this.

[0123] In some embodiments, in order not to affect the user's normal viewing of the playback interface, the progress bar can be set above or below the preview interface.

[0124] Exemplarily, in the case of detecting a trigger operation on a target node on the preview interface, the first thumbnail located at the target node is determined, and the panoramic video reads the first video frame corresponding to the first thumbnail at a predetermined speed, renders the first video frame onto the surface of a sphere through OpenGL, and then maps it from the three-dimensional world to the playback interface of the terminal device through a projection transformation (for example, converting texture information into image information that can be rendered onto the screen through view transformation or mapping transformation, so that the image data can be displayed on the screen).

[0125] In some embodiments, to satisfy the visual coherence of the user, the reading speed of the video frames can be set to more than 20 times per second.

[0126] In the embodiments of the present disclosure, in the case of detecting a trigger operation on a target node on the preview interface, the first thumbnail located at the target node is determined; the first video frame corresponding to the first thumbnail located at the target node is displayed, so that the user can adjust the target node according to the needs and jump to play the first video frame corresponding to the target node, thereby meeting the user's need for accurately positioning the target node and being beneficial to improving the user's viewing experience.

[0127] In some embodiments, the method further includes:

[0128] Detecting a second adjustment operation;

[0129] Determining each third video frame displayed through the visible area of the panoramic video and the second image data of each third video frame; wherein, during the playback of the panoramic video, each third video frame is displayed within the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, and as the panoramic video rotates to different angles, the third video frames within the visible area are different;

[0130] Drawing a fourth video frame based on the second image data corresponding to the third video frame in a pixel buffer;

[0131] Performing compression processing on the fourth video frame in the pixel buffer to obtain a second thumbnail;

[0132] Updating the content displayed on the preview interface based on the second thumbnail.

[0133] As described above, the panoramic video includes scene information within a range of 360 degrees or 720 degrees, and the user can freely select the scene they hope to view according to their actual needs. Therefore, the user can output a second adjustment operation on the panoramic video, thereby forming a new viewing perspective on the panoramic video.

[0134] It can be understood that during the playback of the panoramic video, each third video frame can be displayed within the visible area through the coordinate transformation matrix corresponding to the second adjustment operation. Since only one third video frame can be displayed in the visible area of the panoramic video, when the panoramic video rotates to different angles, the third video frame displayed in the visible area is different.

[0135] It should be explained that after determining the second image data corresponding to each third video frame displayed through the visible area of the panoramic video, based on the second image data, a fourth video frame can be drawn in the pixel buffer, and the fourth video frame in the pixel buffer can be compressed to obtain a second thumbnail.

[0136] In some embodiments, to improve processing efficiency, after drawing the fourth video frame in the pixel buffer based on the second image data corresponding to the third video frame, each fourth video frame can be first stored in the pixel buffer. After receiving the instruction to read to the frame buffer corresponding to the CPU, the drawn fourth video frame is read into the frame buffer (i.e., memory) corresponding to the CPU, and then the format processing operation is performed to obtain a second thumbnail.

[0137] It can be understood that to enhance the user's viewing experience, after generating the second thumbnail, the display content of the first thumbnail on the preview interface is cleared, and each second thumbnail corresponding to the second adjustment operation is redisplayed, enabling the user to view the video content of the second video frame corresponding to the second adjustment operation based on the second thumbnail on the preview interface.

[0138] In some embodiments, a progress bar can be generated for each second thumbnail corresponding to the fourth video frame, facilitating the user to adjust each node in the progress bar to determine the second thumbnail located at the target node. After determining the second thumbnail located at the target node, the second video frame corresponding to the second thumbnail at the target node is then displayed on the playback interface, and then starting from the second video frame, the panoramic video is played.

[0139] In an embodiment of the present disclosure, a second adjustment operation is detected; first, each third video frame displayed in the visible area of the panoramic video and second image data of each third video frame are determined; wherein, during the playback of the panoramic video, each third video frame is displayed in the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, and as the panoramic video rotates to different angles, the third video frames in the visible area are different; then, a fourth video frame is drawn in a pixel buffer based on the second image data corresponding to the third video frame; then, the fourth video frame in the pixel buffer is compressed to obtain a second thumbnail; finally, the preview interface is updated based on the second thumbnail. In this way, when the second adjustment operation is detected in the embodiment of the present disclosure, a second thumbnail corresponding to the second adjustment operation is regenerated and the preview interface is updated, so that the user can arbitrarily adjust the viewing angle of the panoramic image according to needs and obtain the thumbnail corresponding to the viewing angle, thereby improving the user's operation experience.

[0140] In some embodiments, the method further includes:

[0141] If no operation on the panoramic video is detected within a preset duration, it is determined that the first adjustment operation has ended.

[0142] It can be understood that, in order to improve the accuracy of generating the first thumbnail, a preset duration can be set to determine whether the first operation has ended, so as to reduce the misjudgment of the end of the first adjustment operation.

[0143] If no operation on the panoramic video is detected within the preset duration, it is determined that the first adjustment operation has ended; if an operation on the panoramic video is detected within the preset duration, it is determined that the first adjustment operation has not ended.

[0144] Here, the preset duration can be set arbitrarily according to needs, and the embodiment of the present disclosure does not limit this.

[0145] Exemplarily, the preset duration is set to 100 milliseconds. If no sliding operation of the user on the panoramic video is detected within 100 milliseconds, it is determined that the first adjustment operation has ended; if a sliding operation of the user on the panoramic video is detected within 100 milliseconds, it is determined that the first adjustment operation has not ended.

[0146] In the embodiment of the present disclosure, if no operation on the panoramic video is detected within the preset duration, it is determined that the first adjustment operation has ended, which is beneficial to reducing the misjudgment of the end of the first adjustment operation and improving the accuracy of generating the first thumbnail.

[0147] In some embodiments, the method further includes:

[0148] Storing each of the first thumbnails obtained by compressing each of the second video frames into the pixel buffer;

[0149] The displaying the first thumbnail on the preview interface includes:

[0150] Selecting, from each of the first thumbnails, the first thumbnail corresponding to the target moment indicated by the application operation, and displaying the first thumbnail corresponding to the target moment on the preview interface.

[0151] It can be understood that, in order to improve the user experience, each of the first thumbnails obtained by compressing each of the second video frames can be stored in the pixel buffer, so that, according to the user's needs, the first thumbnail meeting the user's needs can be displayed on the preview interface.

[0152] Specifically, each frame of the first video frame corresponds to a thumbnail. Therefore, after detecting the user's application operation, the first thumbnail corresponding to the target moment corresponding to the application operation can be selected from each of the first thumbnails.

[0153] Exemplarily, for a 30-second panoramic video, according to the user's needs, the first thumbnails corresponding to different moments can be selected. For example, if the user selects the first thumbnail corresponding to the 0th second, the first thumbnail corresponding to the 10th second, or the first thumbnail corresponding to the 20th second, the first thumbnail corresponding to the 0th second, the first thumbnail corresponding to the 10th second, or the first thumbnail corresponding to the 20th second can be obtained from the pixel buffer, and the first thumbnail corresponding to the 0th second, the first thumbnail corresponding to the 10th second, or the first thumbnail corresponding to the 20th second is displayed on the preview interface.

[0154] In the embodiments of the present disclosure, each of the first thumbnails obtained by compressing each of the second video frames is stored in the pixel buffer; such that after detecting the user's application operation, the first thumbnail corresponding to the target moment indicated by the application operation is selected from each of the first thumbnails, and the first thumbnail corresponding to the target moment is displayed on the preview interface, thereby facilitating the improvement of the user experience.

[0155] Figure 5 is a schematic diagram of generating a thumbnail corresponding to a panoramic video shown according to an exemplary embodiment, as Figure 5 shown;

[0156] In step 201, spherical rendering.

[0157] In some embodiments, the sphere model is drawn in a virtual environment using OpenGL, and the decoded panoramic video frame D2 is rendered onto the sphere model to form a spherical panoramic video D2.

[0158] In step 202, a coordinate transformation matrix identical to that in the playback process is established.

[0159] In some embodiments, a coordinate transformation matrix d1 is established for the spherical panoramic video D2, and each first video frame displayed through the visible area of the panoramic video and the first image data of each of the first video frames are determined; wherein, during the playback of the panoramic video, each of the first video frames is displayed within the visible area through the coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames within the visible area are different.

[0160] In step 203, the first image data corresponding to the first video frame of the visible area is rendered into the pixel buffer.

[0161] In some embodiments, the first image data corresponding to the first video frame of the visible area is rendered into the pixel buffer, and a second video frame is drawn in the pixel buffer based on the first image data corresponding to the first video frame.

[0162] Exemplarily, to ensure the drawing effect of the second video frame, after obtaining the first image data of the first video frame within the visible area of the panoramic video, the first image data may be preprocessed, and each second video image is drawn in real time in the pixel buffer. Wherein, the preprocessing may be a patterned processing of the video frame, such as modifying the color mode of the video frame, the size of the video frame, or performing filtering, distortion correction, etc.

[0163] In step 204, compression processing is performed.

[0164] In some embodiments, after all the second video frames are drawn, the second video frames in the pixel buffer may be compressed to obtain a first thumbnail, thereby facilitating the user to obtain the video content within the visible area. Exemplarily, the drawn second video image D3 is stored in the pixel buffer, and after receiving the instruction to read to the frame buffer corresponding to the CPU, the drawn second video image D3 is read into the frame buffer (i.e., memory) corresponding to the CPU, and then a compression processing operation is performed to obtain a first thumbnail D4 corresponding to the first adjustment operation.

[0165] Figure 6 is a block diagram of an image processing apparatus shown according to an exemplary embodiment, as Figure 6 shown, the image processing apparatus 400 includes:

[0166] The first determination module 401 is configured to determine each first video frame displayed in the visible area of the panoramic video and the first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, and when the panoramic video rotates to different angles, the first video frames in the visible area are different;

[0167] The first drawing module 402 is configured to draw a second video frame in a pixel buffer based on the first image data corresponding to the first video frame;

[0168] The first execution module 403 is configured to perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display each of the first thumbnails on a preview interface.

[0169] In some embodiments, the first determination module 401 is specifically configured to:

[0170] During the execution of a first adjustment operation on the panoramic video, determine each first video frame displayed in the visible area of the panoramic video and the first image data of each of the first video frames; or

[0171] When the first adjustment operation is completed, determine each first video frame displayed in the visible area of the panoramic video and the first image data of each of the first video frames.

[0172] In some embodiments, the image processing apparatus 400 further includes:

[0173] A second determination module, configured to determine the first thumbnail located at the target node when a trigger operation on the target node on the preview interface is detected;

[0174] A first display module, configured to display the first video frame corresponding to the first thumbnail located at the target node.

[0175] In some embodiments, the image processing apparatus 400 further includes:

[0176] A first detection module, configured to detect a second adjustment operation;

[0177] A third determination module, configured to determine each third video frame displayed through the visible area of the panoramic video and second image data of each of the third video frames; wherein, during the playback of the panoramic video, each of the third video frames is displayed within the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, and as the panoramic video rotates to different angles, the third video frames within the visible area are different;

[0178] A second drawing module, configured to draw a fourth video frame in a pixel buffer based on the second image data corresponding to the third video frame;

[0179] A second execution module, configured to perform compression processing on the fourth video frame in the pixel buffer to obtain a second thumbnail;

[0180] An update module, configured to update the content displayed on the preview interface based on the second thumbnail.

[0181] In some embodiments, the image processing device 400 further includes:

[0182] A fourth determination module, configured to determine that the execution of the first adjustment operation ends if no operation on the panoramic video is detected within a preset duration.

[0183] In some embodiments, the image processing device 400 further includes:

[0184] A storage module, configured to store each of the first thumbnails obtained by compressing each of the second video frames into the pixel buffer;

[0185] The first execution module 403 is specifically configured to:

[0186] Select a first thumbnail at a target moment indicated by an application operation from each of the first thumbnails, and display the first thumbnail at the target moment on the preview interface.

[0187] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0188] Figure 7 is a hardware structure block diagram of an electronic device shown according to an exemplary embodiment Figure 1 . For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0189] Refer to Figure 7, the apparatus 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0190] The processing component 802 generally controls the overall operation of the apparatus 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0191] The memory 804 is configured to store various types of data to support the operation of the apparatus 800. Examples of such data include instructions for any application or method operating on the apparatus 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0192] The power component 806 provides power to the various components of the apparatus 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 800.

[0193] The multimedia component 808 includes a screen that provides an output interface between the apparatus 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the apparatus 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0194] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0195] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0196] The sensor component 814 includes one or more sensors for providing an assessment of the state of the device 800 in various aspects. For example, the sensor component 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor component 814 can also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0197] The communication component 816 is configured to facilitate communication between the device 800 and other devices in a wired or wireless manner. The device 800 can access a wireless network based on communication standards, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0198] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0199] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by the processor 820 of the apparatus 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, and an optical data storage device, etc.

[0200] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute an image processing method, and the method includes:

[0201] Determining each first video frame displayed through a visible area of a panoramic video and first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, the panoramic video rotates to different angles, and the first video frames in the visible area are different;

[0202] Drawing a second video frame in a pixel buffer based on the first image data corresponding to the first video frame;

[0203] Performing compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and displaying each of the first thumbnails on a preview interface.

[0204] Figure 8 is a hardware structure block diagram of an electronic device shown according to an exemplary embodiment Figure 2 . For example, the apparatus 1900 may be provided as a server. Referring to Figure 8 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method:

[0205] Determine each first video frame displayed through the visible area of the panoramic video and the first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed within the visible area through a coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames within the visible area are different;

[0206] Draw a second video frame in the pixel buffer based on the first image data corresponding to the first video frame;

[0207] Perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display each of the first thumbnails on the preview interface.

[0208] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output (I / O) interface 1958. Device 1900 may operate based on an operating system stored in memory 1932, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0209] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0210] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, the method includes: Determining each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed within the visible area through a coordinate transformation matrix, the panoramic video rotates to different angles, and the first video frames within the visible area are different; Drawing a second video frame in a pixel buffer based on the first image data corresponding to the first video frame; Performing compression processing on the second video frame within the pixel buffer to obtain a first thumbnail, and displaying each of the first thumbnails on a preview interface.

2. The method according to claim 1, wherein the determining each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames, includes: During the execution of a first adjustment operation on the panoramic video, determining each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames; or When the execution of the first adjustment operation ends, determining each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames.

3. The method according to claim 1, characterized in that, the method further includes: When a trigger operation on a target node on the preview interface is detected, determining the first thumbnail located at the target node; Displaying the first video frame corresponding to the first thumbnail located at the target node.

4. The method according to claim 1, characterized in that, the method further includes: Detecting a second adjustment operation; Determining each third video frame displayed through the visible area of the panoramic video and second image data of each of the third video frames; wherein, during the playback of the panoramic video, each of the third video frames is displayed within the visible area through a coordinate transformation matrix corresponding to the second adjustment operation, the panoramic video rotates to different angles, and the third video frames within the visible area are different; Drawing a fourth video frame in a pixel buffer based on the second image data corresponding to the third video frame; performing compression processing on the fourth video frame within the pixel buffer to obtain a second thumbnail; Updating the content displayed on the preview interface based on the second thumbnail.

5. The method according to claim 1, characterized in that, the method further includes: If no operation on the panoramic video is detected within a preset duration, determining that the execution of the first adjustment operation ends.

6. The method according to any one of claims 1 to 5, characterized in that, the method further includes: Storing each of the first thumbnails obtained by compressing each of the second video frames into the pixel buffer; The displaying the first thumbnail on the preview interface includes: From each of the first thumbnails, select the first thumbnail at the target moment indicated by the application operation, and display the first thumbnail at the target moment on the preview interface.

7. An image processing apparatus, characterized in that the apparatus includes: a first determination module configured to determine each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames; wherein, during the playback of the panoramic video, each of the first video frames is displayed in the visible area through a coordinate transformation matrix, and as the panoramic video rotates to different angles, the first video frames in the visible area are different; a first drawing module configured to draw a second video frame in a pixel buffer based on the first image data corresponding to the first video frame; a first execution module configured to perform compression processing on the second video frame in the pixel buffer to obtain a first thumbnail, and display each of the first thumbnails on a preview interface.

8. The apparatus according to claim 7, characterized in that the first determination module is specifically configured to: during the execution of a first adjustment operation on the panoramic video, determine each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames; or in the case where the first adjustment operation is completed, determine each first video frame displayed through the visible area of the panoramic video and first image data of each of the first video frames.

9. An electronic device, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to: when executing the executable command, implement the steps in the image processing method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, implement the steps in the image processing method according to any one of claims 1 to 6.

Images