A video playing method and related apparatus

Abstract

The application discloses a video playing method and related device, based on the video to be played, a plurality of video frames can be obtained, based on the current adaptive mode, the size information of the plurality of video frames and the size information of the playing view, the size adaptation information in the current adaptive mode is determined, the size adaptation information includes the position information of the to-be-displayed area in the plurality of video frames and the position information of the mapping area in the texture object corresponding to the playing view, the to-be-displayed area of the plurality of video frames is used for being mapped to the mapping area respectively. Based on the size adaptation information, according to the playing order of the plurality of video frames, the video frame data of the plurality of video frames is loaded to the texture object in turn, the data of the texture object is continuously drawn to the drawing surface object corresponding to the playing view, so as to utilize the playing view to play the video, the size adaptation between the video frame and the playing view is realized, the freedom degree of the size adaptation of the video to be played and the playing view is improved, and meanwhile the superior performance of the playing view is ensured.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a video playback method and related apparatus. Background Technology

[0002] Currently, there is a need for video playback within applications, such as intro videos at application startup, user-created short videos, and movie / TV resource playback. The size of the video to be played is usually fixed; however, the required display size for that video is often variable. For example, different devices may have different screen sizes, and therefore require different display sizes.

[0003] Currently, video playback can be achieved using audio and video playback components. However, these components typically rely on a simple logic to adapt the final width and height based on both the view's width and height and the video's width and height, resulting in limited flexibility in adapting the video's width and height. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a video playback method and related apparatus. Based on the region selection logic defined by the size adaptation information in the current adaptive mode, the method loads the texture of the video frame and draws the texture object, thereby achieving size adaptation between the video frame and the drawing surface object. This improves the freedom of size adaptation between the video to be played and the playback view while ensuring the superior performance of the playback view.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] On one hand, this application provides a video playback method, the method comprising:

[0007] Multiple video frames are obtained based on the video to be played;

[0008] Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object, the size adaptation information under the current adaptive mode is determined; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object;

[0009] Based on the size adaptation information, according to the playback order of the multiple video frames, the video frame data of the multiple video frames are loaded into the texture object in sequence, and the display areas of the multiple video frames are respectively mapped to the mapping area;

[0010] The data of the texture object is continuously drawn onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

[0011] Optionally, determining the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object includes:

[0012] If the current adaptive mode includes full-screen display, then the position information of the mapped region is determined based on the entire area of ​​the texture object;

[0013] Based on the size information of the multiple video frames and the size information of the mapped area, the position information of the area to be displayed is determined, so that the area to be displayed and the mapped area have the same aspect ratio.

[0014] Optionally, determining the location information of the area to be displayed based on the size information of the plurality of video frames and the size information of the mapped area includes:

[0015] According to the current adaptive mode, the position information of the first feature point in the area to be displayed is determined; if the current adaptive mode includes displaying the first vertex side portion, the first feature point is the first vertex, and the position information of the first vertex is determined according to the position information of the first vertices of the multiple video frames; if the current adaptive mode includes displaying the center portion, the first feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center points of the multiple video frames.

[0016] Based on the size information of the multiple video frames and the size information of the mapping area, determine the size of the first side of the area to be displayed and the size of the second side adjacent to the first side;

[0017] The location information of the area to be displayed is determined based on the location information of the first feature point, the size of the first side, and the size of the second side.

[0018] Optionally, determining the size of the first side of the area to be displayed and the size of the second side adjacent to the first side based on the size information of the plurality of video frames and the size information of the mapped area includes:

[0019] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is greater than the aspect ratio of the mapping area, the height of the multiple video frames is used as the size of the first side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the first side, the size of the second side adjacent to the first side is determined.

[0020] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is less than the aspect ratio of the mapping area, the width of the multiple video frames is used as the size of the second side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the second side, the size of the first side adjacent to the second side is determined.

[0021] Optionally, if the current adaptive mode fills the entire screen, the step of loading the video frame data of the multiple video frames sequentially into the texture object based on the size adaptation information and the playback order of the multiple video frames includes:

[0022] Based on the location information of the area to be displayed, the multiple video frames are sequentially extracted according to their playback order to obtain the image to be displayed.

[0023] The image data of the image to be displayed is loaded into the texture object in sequence.

[0024] Optionally, determining the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object includes:

[0025] If the current adaptive mode includes a full content overview display, then the location information of the area to be displayed is determined based on the entire area of ​​the multiple video frames;

[0026] Based on the size information of the area to be displayed and the size information of the texture object, the position information of the mapping area is determined so that the area to be displayed and the mapping area have the same aspect ratio.

[0027] Optionally, determining the position information of the mapped region based on the size information of the area to be displayed and the size information of the texture object includes:

[0028] Based on the current adaptive mode, the position information of the second feature point in the mapping region is determined; if the current adaptive mode includes side display of the second vertex, the second feature point is the second vertex, and the position information of the second vertex is determined according to the position information of the second vertex of the texture object; if the current adaptive mode includes center display, the second feature point is the center point of the region, and the position information of the center point of the region is determined according to the position information of the center point of the texture object.

[0029] Based on the size information of the area to be displayed and the size information of the texture object, determine the size of the third side of the mapping area and the size of the fourth side adjacent to the third side;

[0030] The location information of the mapping region is determined based on the location information of the second feature point, the size of the third side, and the size of the fourth side.

[0031] Optionally, determining the size of the third side of the mapped area and the size of the fourth side adjacent to the third side based on the size information of the area to be displayed and the size information of the texture object includes:

[0032] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is greater than the aspect ratio of the texture object, the width of the texture object is used as the size of the third side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the third side, the size of the fourth side adjacent to the third side is determined.

[0033] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is less than the aspect ratio of the texture object, the height of the texture object is used as the size of the fourth side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the fourth side, the size of the third side adjacent to the fourth side is determined.

[0034] Optionally, the method further includes:

[0035] Add the playback view to the view tree to obtain the drawing surface object corresponding to the playback view; the size information of the drawing surface object is determined according to the size information of the playback view;

[0036] A texture object is created based on the drawing surface object corresponding to the playback view; the size information of the texture object is determined based on the size information of the drawing surface object.

[0037] Optionally, the playback view is a texture view, and the method further includes:

[0038] If a display transformation operation is obtained for the view tree, the display parameters of the playback view are adjusted according to the display transformation operation.

[0039] Optionally, the method further includes:

[0040] During video playback using the playback view, in response to a mode switching operation including a target mode, the current adaptive mode is updated according to the target mode, wherein the target mode is one of the multiple adaptive modes.

[0041] On the other hand, this application provides a video playback device, the device comprising:

[0042] The video frame acquisition unit is used to obtain multiple video frames based on the video to be played.

[0043] The size adaptation information determination unit is used to determine the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object;

[0044] The texture loading unit is used to load the video frame data of the multiple video frames into the texture object sequentially according to the playback order of the multiple video frames based on the size adaptation information, and the display areas of the multiple video frames are respectively mapped to the mapping area;

[0045] The texture drawing unit is used to continuously draw the data of the texture object onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

[0046] Optionally, the size adaptation information determination unit includes:

[0047] The first position determination unit is used to determine the position information of the mapped region based on the entire region of the texture object if the current adaptive mode includes full-screen display.

[0048] The second position determination unit is used to determine the position information of the area to be displayed based on the size information of the plurality of video frames and the size information of the mapping area, so that the area to be displayed and the mapping area have the same aspect ratio.

[0049] Optionally, the second position determining unit includes:

[0050] The first feature point position determination unit is used to determine the position information of the first feature point in the area to be displayed according to the current adaptive mode; if the current adaptive mode includes displaying the first vertex side portion, the first feature point is the first vertex, and the position information of the first vertex is determined according to the position information of the first vertices of the multiple video frames; if the current adaptive mode includes displaying the center portion, the first feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center points of the multiple video frames.

[0051] The first size determination unit is used to determine the size of the first side of the area to be displayed and the size of the second side adjacent to the first side based on the size information of the plurality of video frames and the size information of the mapping area;

[0052] The area to be displayed subunit is used to determine the location information of the area to be displayed based on the location information of the first feature point, the size of the first side and the size of the second side.

[0053] Optionally, the first size determination unit is specifically used for:

[0054] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is greater than the aspect ratio of the mapping area, the height of the multiple video frames is used as the size of the first side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the first side, the size of the second side adjacent to the first side is determined.

[0055] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is less than the aspect ratio of the mapping area, the width of the multiple video frames is used as the size of the second side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the second side, the size of the first side adjacent to the second side is determined.

[0056] Optionally, if the current adaptive mode fills the entire screen, the texture loading unit includes:

[0057] An adaptive module is used to extract the image to be displayed by sequentially capturing the multiple video frames according to the playback order of the multiple video frames, based on the location information of the area to be displayed.

[0058] The texture loading subunit is used to sequentially load the image data of the image to be displayed into the texture object.

[0059] Optionally, the size adaptation information determination unit includes:

[0060] The third position determination unit is used to determine the position information of the area to be displayed based on the entire area of ​​the plurality of video frames if the current adaptive mode includes a full view display.

[0061] The fourth position determination unit is used to determine the position information of the mapping area based on the size information of the area to be displayed and the size information of the texture object, so that the area to be displayed and the mapping area have the same aspect ratio.

[0062] Optionally, the fourth position determination unit includes:

[0063] The second feature point position determination unit is used to determine the position information of the second feature point in the mapping area according to the current adaptive mode; if the current adaptive mode includes second vertex side display, the second feature point is the second vertex, and the position information of the second vertex is determined according to the position information of the second vertex of the texture object; if the current adaptive mode includes center display, the second feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center point of the texture object.

[0064] The second size determination unit is used to determine the size of the third side of the mapping area and the size of the fourth side adjacent to the third side based on the size information of the area to be displayed and the size information of the texture object.

[0065] The mapping region determination subunit is used to determine the location information of the mapping region based on the location information of the second feature point, the size of the third side, and the size of the fourth side.

[0066] Optionally, the second size determination unit is specifically used for:

[0067] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is greater than the aspect ratio of the texture object, the width of the texture object is used as the size of the third side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the third side, the size of the fourth side adjacent to the third side is determined.

[0068] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is less than the aspect ratio of the texture object, the height of the texture object is used as the size of the fourth side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the fourth side, the size of the third side adjacent to the fourth side is determined.

[0069] Optionally, the device further includes:

[0070] A playback view adding unit is used to add the playback view to the view tree to obtain the drawing surface object corresponding to the playback view; the size information of the drawing surface object is determined according to the size information of the playback view;

[0071] The texture object creation unit is used to create a texture object based on the drawing surface object corresponding to the playback view; the size information of the texture object is determined based on the size information of the drawing surface object.

[0072] Optionally, the playback view is a texture view, and the device further includes:

[0073] The display parameter adjustment unit is used to adjust the display parameters of the playback view according to the display transformation operation if a display transformation operation for the view tree is obtained.

[0074] Optionally, the device further includes:

[0075] An adaptive mode switching unit is configured to, during video playback using the playback view, update the current adaptive mode according to a target mode in response to a mode switching operation including a target mode, wherein the target mode is one of the plurality of adaptive modes.

[0076] On the other hand, this application provides a computer device, the device including a processor and a memory:

[0077] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0078] The processor is configured to execute the video playback method described above according to instructions in the computer program.

[0079] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for executing the video playback method described above.

[0080] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to execute the video playback method.

[0081] As can be seen from the above technical solution, multiple video frames can be obtained based on the video to be played. Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the playback view, the size adaptation information under the current adaptive mode can be determined. The size information of the texture object is determined according to the size of the playback view corresponding to the texture object. The current adaptive mode is one of several adaptive modes. The size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object corresponding to the playback view. The areas to be displayed in the multiple video frames are used to map to the mapped areas respectively. It can be understood that when the area to be displayed is smaller than the overall area of ​​the video frame, the mapping process is equivalent to truncating multiple video frames; when the mapped area is smaller than the overall area of ​​the texture object, the mapping process is equivalent to selecting a region of the texture object. Therefore, through the size adaptation information under the current adaptive mode, the region selection logic of the video frame and the texture object under the current adaptive mode can be defined to achieve size adaptation between the video frame and the texture object under the current adaptive mode.

[0082] Then, based on the size adaptation information, the video frame data of multiple video frames can be loaded into the texture object in sequence according to the playback order of multiple video frames, and the data of the texture object can be continuously drawn onto the drawing surface object corresponding to the playback view so as to use the playback view for video playback. In other words, based on the region selection logic defined by the size adaptation information in the current adaptive mode, texture loading and texture object drawing of video frames can be performed. That is, by adapting the size of video frames and texture objects, size adaptation between video frames and the playback view is achieved, so that the area to be displayed in the video frame can be displayed in the video playback area corresponding to the mapped area in the playback view. In this way, the size adaptation problem between the video to be played and the playback view is transformed into the size adaptation between video frames and texture objects during texture loading. Different adaptive modes can correspond to different size adaptation logics for video frames and texture objects, greatly improving the freedom of size adaptation between the video to be played and the playback view. Compared with the size adaptation between the video to be played and the playback view, the adaptation between image frames and texture objects is more efficient and has better real-time performance. Size adaptation can be achieved without adjusting the playback view during playback, without generating additional memory consumption. In this way, while improving the freedom of size adaptation between the video to be played and the playback view, the superior performance of the playback view is guaranteed. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 A schematic diagram illustrating an application scenario of a video playback method provided in an embodiment of this application;

[0085] Figure 2 This application provides a schematic diagram of a video playback process as an embodiment of the present application.

[0086] Figure 3 This is another schematic diagram of a video playback process provided in an embodiment of this application;

[0087] Figure 4 A flowchart illustrating a video playback method provided in this application embodiment;

[0088] Figure 5 A screenshot illustrating a video playback method provided in an embodiment of this application;

[0089] Figure 6 A schematic diagram of a video playback framework provided in an embodiment of this application;

[0090] Figure 7 , Figure 8 and Figure 9 This is a schematic diagram illustrating various implementation methods of full-screen display in the embodiments of this application;

[0091] Figure 10 and Figure 11 This is a schematic diagram illustrating various implementation methods that provide an overview of the content in the embodiments of this application;

[0092] Figure 12 This is a schematic diagram of another video playback framework provided in an embodiment of this application;

[0093] Figure 13 A structural block diagram of a video playback device provided in an embodiment of this application;

[0094] Figure 14 A structural diagram of a terminal device provided in an embodiment of this application;

[0095] Figure 15 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0096] The embodiments of this application will now be described with reference to the accompanying drawings.

[0097] Currently, video playback can be achieved using audio and video playback components. However, these components typically rely on a simple logic to adapt the final width and height based on both the view's width and height and the video's width and height, resulting in limited flexibility in adapting the video's width and height.

[0098] To address the aforementioned technical issues, this application provides a video playback method and related apparatus. Based on the region selection logic defined by the size adaptation information in the current adaptive mode, the method loads the texture of the video frame and draws the texture object, thereby achieving size adaptation between the video frame and the drawing surface object. This improves the freedom of size adaptation between the video to be played and the playback view while ensuring the superior performance of the playback view.

[0099] The video playback method provided in this application can be implemented using a computer device, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0100] To facilitate understanding of the technical solutions provided in this application, a video playback method provided in this application embodiment will be introduced next in conjunction with a practical application scenario.

[0101] Figure 1 This illustration shows an application scenario of a video playback method provided in an embodiment of this application. The scenario includes a server 20 and a terminal device 10. The terminal device 10 has an application installed for video playback. The server 20 and the terminal device 10 interact via a network. The terminal device 10 may have a playback area 100. The server 20 or the terminal device 10 can function as a computer device, used to perform video playback operations according to a video playback task. The terminal device 10 interacts with the user, obtains the video playback task, and displays the video in the playback area 100. The following description uses the terminal device 10 as an example of a computer device.

[0102] Terminal device 10 can obtain multiple video frames based on the video to be played. Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the playback view, it can determine the size adaptation information under the current adaptive mode. The size information of the texture object is determined according to the size of the playback view corresponding to the texture object. The current adaptive mode is one of several adaptive modes. The size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object corresponding to the playback view. The areas to be displayed in the multiple video frames are used to map to the mapped areas respectively. It can be understood that when the area to be displayed is smaller than the overall area of ​​the video frame, the mapping process is equivalent to truncating multiple video frames; when the mapped area is smaller than the overall area of ​​the texture object, the mapping process is equivalent to selecting a region of the texture object. Therefore, through the size adaptation information under the current adaptive mode, the region selection logic of the video frame and texture object under the current adaptive mode can be defined to achieve size adaptation between the video frame and texture object under the current adaptive mode.

[0103] After terminal device 10, based on size adaptation information, it can load the video frame data of multiple video frames into a texture object in sequence according to the playback order of multiple video frames, and then continuously draw the data of the texture object onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback. In other words, based on the region selection logic defined by the size adaptation information in the current adaptive mode, texture loading and texture object drawing of video frames can be performed. That is, by adapting the size of video frames and texture objects, size adaptation between video frames and the playback view is achieved, so that the area to be displayed in the video frame can be displayed in the video playback area corresponding to the mapped area in the playback view. In this way, the size adaptation problem between the video to be played and the playback view is transformed into the size adaptation between video frames and texture objects during texture loading. Different adaptive modes can correspond to different size adaptation logics for video frames and texture objects, greatly improving the freedom of size adaptation between the video to be played and the playback view. Compared with the size adaptation between the video to be played and the playback view, the adaptation between image frames and texture objects is more efficient and has better real-time performance. Size adaptation can be achieved without adjusting the playback view during playback, without generating additional memory consumption. In this way, while improving the freedom of size adaptation between the video to be played and the playback view, the superior performance of the playback view is guaranteed.

[0104] For ease of explanation, the relevant concepts involved in the embodiments of this application are introduced below.

[0105] Terminal devices can have an operating system. Android is an open-source operating system developed by Google, based on the Linux kernel, and primarily used in mobile devices such as smartphones and tablets. Since its initial release in 2008, Android has become one of the most widely used mobile operating systems in the world. Android boasts many features, such as a vast application ecosystem, a user-friendly interface, regular version releases bringing new features and security improvements, and powerful development tools and resources to support Android application development. Crucially, Android is open-source, meaning developers are free to view, modify, and distribute its source code. This openness makes Android highly flexible, allowing for extensive customization and application across a wide range of devices.

[0106] The operating system provides views for developers to create user interfaces. In Android development, the native View framework is the foundation for building user interfaces. Android provides a rich set of View and ViewGroup classes for creating and managing user interface components. View is the base class for all user interface (UI) components in Android, and it is the basic element that constitutes the user interface. It represents a rectangular area on the screen, where content is drawn and user interactions are handled; for example, buttons and text boxes are subclasses of View. ViewGroup is a special type of View that can contain other Views (including other ViewGroups). ViewGroup is the base class for all layout classes, and common subclasses include LinearLayout, RelativeLayout, FrameLayout, and ConstraintLayout.

[0107] Views can include playback views, used for multimedia playback, such as playing still images, moving images, or videos. In Android, playback views can include TextureView, SurfaceView, and VideoView, all of which are subclasses of View provided by the Android system. The following section introduces these three types of playback views and other related concepts.

[0108] First, TextureView is a view class provided by Android for displaying dynamic images. Essentially, it's a regular View that can be added to the view hierarchy just like any other View. TextureView provides an independent drawing surface object. Developers can use the graphics rendering API (Application Programming Interface), video decoding, and other operations to fill this drawing surface object with drawing content, thus enabling dynamic image playback using TextureView. TextureView doesn't create a separate window; instead, it draws within the regular View hierarchy, allowing it to be transformed, animated, and clipped together with other Views.

[0109] Surface is a crucial class in Android development. It's an abstract drawing surface that provides a buffer for direct drawing. Drawing operations can be performed on this surface, providing a concrete carrier for higher-level drawing operations and offering low-level drawing support for the View. Surface is typically used in scenarios requiring direct access to the display buffer, such as video playback, game rendering, and camera preview. The Surface class itself doesn't perform any drawing operations; instead, it provides an interface that allows other components (such as Canvas, OpenGL, or MediaCodec) to draw on it.

[0110] In the process of drawing a user interface using a Surface, the user interface can be mapped to a Surface object, and its content can be displayed within the Surface. Essentially, the content of the user interface is filled by drawing on the Surface object. Taking a View in the user interface as an example, the drawing process of a View ultimately takes place on the Surface. When a View needs to display content, it performs drawing operations through the canvas provided by the Surface, drawing the appearance and content of the View onto the memory area represented by the Surface. Then, the system displays the content of the Surface on the screen.

[0111] Other components, such as OpenGL, are cross-platform graphics APIs designed for embedded systems like mobile devices, tablets, and game consoles. They provide efficient 2D and 3D graphics rendering capabilities. OpenGL is widely used in mobile operating systems like Android and iOS for developing high-performance graphics applications and games. It consists of a series of functions and tools that provide developers with powerful graphics processing capabilities, enabling them to create highly realistic and interactive graphics applications. In OpenGL, a texture is image data used to map onto a 3D graphics surface. Texture objects are used to store and manage the corresponding texture data. Textures can greatly enhance the visual effects of graphics, making rendered objects look more realistic and detailed. Surfaces can be used as OpenGL rendering objects. OpenGL provides a rich API for loading, binding, and manipulating texture data. After image data is uploaded to a texture ID, it becomes the texture data corresponding to that texture ID. This ID can be used to manipulate the texture data to transform the image or draw it onto the screen.

[0112] Other components, such as MediaCodec, are lower-level audio and video codec APIs provided by the Android platform for accessing underlying media codec functions. They offer the ability to encode and decode audio and video data, enabling developers to implement efficient media processing in Android applications. Through MediaCodec, developers can encode raw audio or video data according to specific encoding formats for storage or transmission, and can also decode received encoded data to restore it to playable audio or video streams, thereby enabling video playback, recording, live streaming, and other functions. MediaCodec is implemented in C++ and can directly use hardware codecs, providing efficient audio and video processing capabilities and greatly improving video processing efficiency. MediaCodec supports various common audio and video formats, such as H.264, AAC, and MP4. MediaCodec has low latency, making it suitable for real-time audio and video processing. Furthermore, MediaCodec is also suitable for use in custom audio and video decoding and rendering players.

[0113] Secondly, SurfaceView is also a view class in Android used to display graphics and video. It provides an independent drawing surface (Surface) object, allowing applications to perform drawing operations on a separate drawing surface. SurfaceView draws by creating an independent window, so it can draw in a background thread without affecting the performance of the main thread, resulting in high drawing performance. This makes SurfaceView ideal for scenarios requiring high-performance drawing, such as video playback, game rendering, and camera preview, enabling real-time updates and high frame rate display. Unlike TextureView, SurfaceView cannot be used with other views. Figure 1 It performs transformations, animations, and cropping.

[0114] In Android, the main thread, also known as the UI thread, is responsible for handling user interface operations, events, and managing other components. User interface operations include updating views and UI actions such as clicks and touches (user input events). Other components include Activities and Services. An Activity is the basic component in an Android application used to implement the user interface, providing a container and context for Views and ViewGroups. An Activity can include one or more Views, which together constitute the Activity's user interface. For example, a login Activity might contain multiple Views such as a username input field, a password input field, and a login button.

[0115] Secondly, VideoView is a view class provided by Android specifically for playing videos. As an audio and video playback component in Android, it is mainly used in video playback scenarios. It is a component that inherits from Android SurfaceView and is a wrapper around SurfaceView. VideoView encapsulates the basic functions of video playback, allowing developers to easily integrate video playback functionality into their applications without having to handle the underlying media decoding and rendering details, thus simplifying video playback operations. Essentially, VideoView is a View that can be added directly to the layout file like a regular View and combined with other Views. Its display position and size are determined by the layout parameters. VideoView integrates a media player (MediaPlayer) internally, which can directly play local video files or network video streams. VideoView is suitable for simple video playback scenarios. For more complex video processing needs, MediaPlayer, SurfaceView, or TextureView may be used.

[0116] MediaPlayer is a class provided by Android for playing audio and video. It offers complete media playback functionality and is the core underlying API for video playback in the Java layer of the Android platform. The video playback capabilities encapsulated by VideoView are implemented based on MediaPlayer. MediaPlayer itself does not have display functionality; it is usually used in conjunction with view components such as SurfaceView or TextureView to display the video image. Developers need to associate MediaPlayer with these view components and render the video data onto the corresponding views. While this approach increases development complexity, it also provides greater flexibility, such as enabling custom video rendering effects.

[0117] On the Android platform, the simplest way to play videos is to directly use the system-provided VideoView component. VideoView is a highly encapsulated version of MediaPlayer, making video playback very convenient for users. (Reference) Figure 2 The diagram shown is a schematic of a video playback process provided in an embodiment of this application. VideoView inherits from SurfaceView and integrates a media player (MediaPlayer) and a media codec (MediaCodec). The video to be played is encoded and decoded by calling MediaCodec through the VideoView component, and played by calling MediaPlayer, thereby outputting the video screen.

[0118] However, in this playback method, VideoView has its own internal logic for final width and height adaptation based on its own width and height and the video's width and height. Developers cannot customize or extend this logic; they can only use the width and height adaptation mode defined by VideoView. In this logic, the video's width and height are typically scaled proportionally until the width or height meets the VideoView's preset dimensions, leaving black borders for the remaining video area. Alternatively, the VideoView's width and height are set to match the video's dimensions, and the VideoView's dimensions are scaled proportionally to fit the drawable area. In other words, VideoView prioritizes preventing the video itself from being stretched or compressed, while simultaneously scaling the video or VideoView proportionally until the width or height fills the VideoView's drawable area. Whether black borders are left or how much video content is preserved during scaling is determined by the single, pre-defined width and height adaptation logic within VideoView.

[0119] Another approach to video playback uses a custom TextureView as the video playback container. The width and height of the TextureView are adjusted to fit the video, allowing for customized width and height adaptation. (See reference.) Figure 3 The diagram illustrates another video playback process provided in this application embodiment. The TextureView provides a surface to the underlying layer, allowing the media player to draw on it to achieve video playback. During playback, the MediaCodec encodes and decodes the video to be played, and the MediaPlayer provides a video playback API to the TextureView, thereby displaying the video image through the TextureView. In this mode, the content provided by the media player always fills the entire TextureView (covering the entire Surface). If the width and height of the TextureView match the width and height of the video, the video image is not stretched or compressed; otherwise, the video image is stretched and compressed.

[0120] The TextureView contains a width and height calculation module. This module calculates and adjusts the TextureView's width and height according to the desired adaptation mode to ensure that the TextureView's width and height match the video's aspect ratio. It then scales the TextureView to meet actual playback requirements, thus guaranteeing a smooth video playback experience. For example, in common full-screen playback, the TextureView's width and height need to be scaled according to the video's aspect ratio until the TextureView completely fills the screen.

[0121] However, in certain video footage scenarios, the scaling ratio of the TextureView itself can be quite large. For example, in landscape mode video playback, the TextureView is adapted to fill the screen. When the video footage is portrait mode, the TextureView needs to be stretched several times to fill the screen. Because the TextureView itself is large, it contains a large number of pixels, leading to significant memory consumption. Furthermore, this approach cannot effectively handle switching between different adaptation modes (it requires recalculating and setting the width and height of the TextureView, causing screen jitter).

[0122] Figure 4 This is a flowchart of a video playback method provided in an embodiment of the present application. In this embodiment, a terminal device is used as the aforementioned computer device for description. The video playback method may include the following steps S101-S104.

[0123] S101, obtains multiple video frames based on the video to be played.

[0124] In this embodiment, the video to be played refers to a video that needs to be played in an application, such as a self-made short video, a movie or TV show resource, or an intro video when the application starts. The application can be an application on various operating system platforms, such as an application on the Android platform. (See reference...) Figure 5 The image shown is a screenshot illustrating a video playback method provided in an embodiment of this application. The video is the intro video played when the application starts, and it fills the entire display screen.

[0125] Playback of the video to be played requires matching with the playback area of ​​the terminal device. The playback area refers to the area on the terminal device's display screen used for playing the video. When playing video through a playback view, the playback area is the area on the screen corresponding to the visible area of ​​the playback view, and the size of the visible area of ​​the playback view is the same as the size of the playback area. It should be noted that this article does not consider the invisible area of ​​the playback view. The size of the visible area of ​​the playback view is simply described as the size of the playback view. If there is also an invisible area in the playback view, the total size of the playback view is the sum of the aforementioned size of the playback view and the size of the invisible area.

[0126] The size of the playback area (visible area) is related to the screen size of the terminal device and also to the playback requirements in different scenarios, such as full-screen display or centered display. In full-screen display, the playback area is the entire display screen area of ​​the terminal device. Due to the diversity of terminal device screen sizes and playback requirements, the size of the playback area and the video to be played are often different. Adapting the size of the video to be played to the playback area is beneficial for improving display quality, addressing these complex scenarios of different terminal devices and playback requirements. How the video content is mapped onto the playback area is the main problem that video player adaptive width and height functionality solves.

[0127] Before playing a video, multiple video frames can be obtained based on the video. These frames often have the same aspect ratio and a specific playback order. During playback, multiple video frames can be continuously acquired for real-time subsequent operations, ultimately enabling video playback. Compared to using platform-encapsulated video playback APIs, this embodiment requires development based on underlying audio and video interfaces to achieve highly customized playback functionality. Specifically, the video to be played can be decoded to obtain multiple video frames. For example, decoding logic can be executed using a media codec, which can be implemented using the MediaCodec module on the Android platform.

[0128] refer to Figure 6 The diagram shown is a schematic of a video playback framework provided in an embodiment of this application. The video playback framework may include a player upper layer and a media codec. The player upper layer obtains the video to be played. The media codec, such as the MediaCodec module of the Android platform, is used to decode the video to be played to obtain multiple video frames and continuously outputs the video frame data of multiple video frames for use. The output order of the video frame data of multiple video frames may be the same as the playback order of multiple video frames, or it may be slightly different.

[0129] Video frame data, consisting of multiple video frames, is typically represented in formats such as YUV or RGB. These video frames contain size information, which may include the width and height of the frames, denoted as w and w respectively. v and h v .

[0130] S102, based on the current adaptive mode, the size information of multiple video frames and the size information of the texture object, determine the size adaptation information under the current adaptive mode.

[0131] In this embodiment, the current adaptive mode can be determined. This current adaptive mode is one of multiple adaptive modes, which can be a preset adaptive mode or the adaptive mode corresponding to a mode switching operation. Multiple adaptive modes are built into the playback component, enabling rapid application and switching of adaptive modes. From the perspective of screen filling, the multiple adaptive modes can include both full-screen display and content overview display; different screen filling angles can achieve different visual effect requirements. Of course, the multiple adaptive modes can also include a custom mode, which can directly define corresponding size adaptation information; distance details are not provided here.

[0132] In practice, if no mode switching operation is received, the current adaptive mode can be the preset adaptive mode, i.e., the default adaptive mode. If a mode switching operation is received during video playback, and the operation corresponds to a target mode, the current adaptive mode can be updated according to the target mode, which is also one of several adaptive modes. By building multiple adaptive modes, video playback under various adaptive modes can be achieved, and switching between different adaptive modes can be performed, increasing the flexibility of size adaptation. The current adaptive mode and the target mode can be the same or different. If the current adaptive mode and the target mode are the same, the current adaptive mode is either not updated or updated to the target mode, and the adaptive mode remains unchanged. If the current adaptive mode and the target mode are different, the current adaptive mode is updated to the target mode, meaning the adaptive mode has changed.

[0133] Among them, full-screen display means that the entire playback area is filled with video images. If the size of the original video image is different from the size of the playback area, the size of the original video image can be scaled proportionally until the video image can fill the playback area. If the playback area is the entire screen, the video image fills the entire screen, that is, the video image is displayed in full screen. At this time, the size of the playback view is also the size of the entire screen.

[0134] refer to Figure 7 , Figure 8 and Figure 9 The diagram shown illustrates various implementation methods for full-screen display in embodiments of this application. (Refer to...) Figure 7 As shown, if the original video frame and the playback area have the same aspect ratio, the video frame can be scaled proportionally to perfectly fill the playback area; see reference. Figure 8 and Figure 9 As shown, if the aspect ratio of the original video frame is different from that of the playback area, the video frame can be scaled proportionally and then cropped. The portion of the video frame retained after cropping is the same size as the playback area, thus filling the playback area with the cropped portion.

[0135] Full view display means that after the video image is scaled up, its complete content must be displayed within the playback area. Therefore, when the original video image size and the playback area size are different, blank spaces can be left on the left, right, top, or bottom. These blank spaces are displayed as black borders in the video playback scene. If the playback area is the entire screen, it is divided into a first area with the image and a second area without the image. The first area corresponds to the video playback area in the playback view, and the second area corresponds to the pixel-free area in the playback view, which is displayed as a black area.

[0136] refer to Figure 10 and Figure 11 The diagram shown illustrates various implementation methods for displaying the overall content in the embodiments of this application. Figure 7 The above can also be used as a way to display the full content overview; see reference. Figure 7 As shown, if the aspect ratio of the original video frame is the same as that of the playback area, the video frame can be scaled proportionally to fill the playback area without losing video quality; see reference. Figure 10 and Figure 11 As shown, if the aspect ratio of the original video frame is different from that of the playback area, the video frame can be scaled proportionally and then mapped entirely onto the first area of ​​the playback area, leaving the space outside the first area as the second area.

[0137] A full-screen display can include at least two types of displays: one showing the area on the first vertex side and the other showing the center area. The first vertex side can include the top-left vertex side, the bottom-left vertex side, the top-right vertex side, and the bottom-right vertex side. (Reference) Figure 8 As shown, if the aspect ratio of the video frame is greater than that of the playback area, the entire height of the video frame can be displayed to retain more video content. However, cropping occurs in the width direction. Displaying the top-left and bottom-left vertices actually displays the left portion minus the right portion; similarly, displaying the bottom-right and top-right vertices displays the right portion minus the left portion. Displaying the center portion removes both the left and right portions simultaneously. (Reference) Figure 9 As shown, if the aspect ratio of the video frame is smaller than that of the playback area, the entire width of the video frame can be displayed to retain more video content. If the height is cropped, displaying the top left vertex side actually displays the top portion minus the bottom portion; displaying the bottom right vertex side actually displays the bottom portion minus the top portion; and displaying the center portion removes both the top and bottom portions. Of course, in other implementations where retaining more video content is not necessary, the video frame can be cropped in both width and height, with each first vertex side corresponding to a different area of ​​the video frame to be displayed.

[0138] A comprehensive content display can include at least two methods: display from the second vertex side and centered display. The second vertex side includes the top-left vertex side, bottom-left vertex side, top-right vertex side, and bottom-right vertex side. (See reference) Figure 10 As shown, if the aspect ratio of the video frame is greater than that of the playback area, the width of the playback area can be filled to preserve a larger video playback area. Therefore, the top left and top right vertices are actually displayed with the top side showing the content and the bottom side empty; the bottom right and bottom left vertices are actually displayed with the bottom side showing the content and the top side empty; center display actually has empty space on both the top and bottom sides. (Reference) Figure 11 As shown, if the aspect ratio of the video frame is smaller than that of the playback area, the height of the playback area can be filled to retain a larger video playback area. Therefore, the display on the top-left and bottom-left vertices actually shows the left side with the right side empty, while the display on the bottom-right and top-right vertices actually shows the right side with the left side empty. Centered display actually has empty space on both the left and right sides. Of course, in other implementations where a large video playback area is not required, both the width and height of the playback area can be selected, and the display on each first vertex side corresponds to the display of the video frame in different first areas of the playback area.

[0139] Before video playback, a playback view can be added to the view tree, providing a corresponding drawing surface object. Based on this surface object, a texture object is created. The size of the drawing surface object is determined by the size of the corresponding playback view; for example, the size of both the drawing surface and texture objects are determined by the size of the drawing surface object. The texture object stores texture data and has an object identifier (ID). This ID allows manipulation and retrieval of the texture object's stored texture data. Thus, obtaining the playback view, the drawing surface object provided by the playback view, and the texture object created based on it before video playback essentially constructs the framework for video playback, providing the foundation for the playback of the video.

[0140] The playback view is a view that provides drawing surface objects, which can be a texture view or a surface view. Texture objects can be, for example, OpenGL texture objects, meaning that OpenGL rendering logic is introduced during video playback. (See reference...) Figure 6 As shown, video frame data can be loaded into a texture object using an OpenGL thread. The video frame data then serves as the texture data corresponding to the texture object, also known as a texture image. Texture objects can be created using OpenGL functions.

[0141] Texture objects can have size information, which determines the size of the texture image. The size information includes the width and height of the texture object, denoted as w, respectively. s and h s Texture objects and playback views can have the same aspect ratio, and their dimensions can be identical or proportional. The size information of the playback view corresponding to the texture object can include width and height, denoted as w respectively. t and h t .

[0142] In practice, the aspect ratios of multiple video frames are usually different from those of the playback area. When playing through a playback view and the size of the playback view is equal to the size of the playback area, the aspect ratios of multiple video frames are usually different from those of the playback view. Therefore, the aspect ratios of multiple video frames are also different from those of the texture object. How multiple video frames are mapped to the texture object determines how they are mapped into the playback area. Thus, different adaptive modes can correspond to different size adaptation information.

[0143] Specifically, the size adaptation information for the current adaptive mode can be determined based on the current adaptive mode, the size information of multiple video frames, and the size information of the texture object. This size adaptation information includes the position information of the areas to be displayed in the multiple video frames and the position information of the mapped areas in the texture object. The areas to be displayed in the multiple video frames are mapped to the mapped areas respectively. Different adaptive modes may have corresponding methods for determining the size video information; the process for determining the size adaptation information under different adaptive modes will be explained later.

[0144] It's understandable that when the area to be displayed is smaller than the overall area of ​​the video frame, the mapping process is equivalent to cropping multiple video frames separately. Similarly, when the mapped area is smaller than the overall area of ​​the texture object, the mapping process is equivalent to selecting a region of the texture object. Therefore, the region selection logic for the video frame and texture object in the current adaptive mode is defined by the size adaptation information. The size ratio between the two determines the scaling method of multiple video frames, thus achieving size adaptation between the video frame and texture object in the current adaptive mode.

[0145] In this embodiment, a width-to-height adaptive module can also be provided to determine the size matching information in the current adaptive mode. (See reference) Figure 12 The diagram shown is a schematic of another video playback framework provided in an embodiment of this application. A width and height adaptive module can be set in the OpenGL thread to solve the problem of video width and height adaptation. It can determine the size matching information in the current adaptive mode for use by the OpenGL thread during texture loading.

[0146] The width and height adaptive module also provides an interface for the player's upper layer to select the adaptive mode. Users can switch between adaptive modes via this interface and determine the size adaptation information under the current adaptive mode based on the mode switching operation. After the current adaptive mode is updated, the next frame's playback will take effect. This means that the adaptive mode can be dynamically adjusted in real time during video playback. This playback component gains the ability to dynamically adjust the width and height adaptive mode, enabling interactive video playback and dynamically modifying the adaptive mode based on user behavior during playback.

[0147] The mode switching operation can include triggering operations on the playback area. Different mode switching operations can correspond to different target modes, and the correspondence between mode switching operations and target modes can be set according to the actual situation. Mode switching operations include, for example, triggering operations on the mode switching control, triggering operations on the first area of ​​the playback area containing video, or triggering operations on the second area of ​​the playback area without video, etc.

[0148] For example, the trigger operation on the first area of ​​the playback area can correspond to the adaptive mode of displaying the entire content, and the trigger operation on the second area of ​​the playback area without video can correspond to the adaptive mode of displaying the entire screen; or, the first trigger operation on the first area of ​​the playback area with video can correspond to switching the display of the upper left corner to the display of the center part, and the second trigger operation can correspond to switching the display of the center part to the display of the lower right corner, etc.

[0149] S103, based on size adaptation information, loads the video frame data of multiple video frames into the texture object sequentially according to the playback order of multiple video frames.

[0150] After determining the size adaptation information, the video frame data of multiple video frames can be loaded into a texture object sequentially according to the playback order. The video frame data loaded into the texture object serves as the texture data corresponding to the texture object, also known as the texture image or texture object data. Specifically, the video frame data of multiple video frames can be loaded into the memory space corresponding to the texture object sequentially. In this way, the video frames loaded into the texture object can perform subsequent operations for display. The display areas of multiple video frames are used to map to mapping areas respectively. When the display area is smaller than the overall area of ​​the video frames, the mapping process is equivalent to truncating multiple video frames. When the mapping area is smaller than the overall area of ​​the texture object, the mapping process is equivalent to selecting a region of the texture object.

[0151] Loading video frame data into a texture object can be achieved using functions, such as the `glTexImage2D` function. During this process, a mapping region and a display region can be defined using functions. Alternatively, based on the position information of the display region, multiple video frames can be captured sequentially according to their playback order to obtain images to be displayed, and then their image data can be loaded into the texture object sequentially. In this case, a mapping region can be defined using a function, with the display image used to define the display region. Alternatively, the display region can be defined using a function, with the mapping region defaulting to the entire texture object. Furthermore, if the mapping region defaults to the entire texture object, and the display region is either the entire area of ​​the display image or the entire area of ​​the video frame, then the function does not need to define the mapping region and the display region. Capturing images from multiple video frames can be implemented using a width-height adaptive module, where the image data of the display image is called and loaded into the texture object, simplifying the parameter configuration of the texture loading process.

[0152] Before loading video frame data from multiple video frames into a texture object, the video frame data can be converted to a texture data format suitable for OpenGL processing. This may involve operations such as color space conversion and data format adjustment, for example, converting YUV format to RGB format and arranging the data into a texture image format that OpenGL can recognize.

[0153] S104, continuously draw the data of the texture object onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

[0154] In this embodiment, after loading the video frame data of the video frame into the texture object, the data of the texture object can be continuously drawn onto the drawing surface object corresponding to the playback view. This is equivalent to using the texture image of the texture object as a texture map of the drawing surface object, so that the playback view has video content. When multiple video frames are loaded into the texture object in the order of playback, the playback view displays the content of the area to be displayed for each video frame in sequence, thus realizing the playback of the video to be played.

[0155] In other words, based on the region selection logic defined by the size adaptation information in the current adaptive mode, texture loading and texture object drawing of video frames can be performed. That is, by adapting the size of the video frame and the texture object, the size adaptation between the video frame and the drawing surface object is achieved, so that the area to be displayed in the video frame can be displayed in the video playback area corresponding to the mapped area in the playback view. This transforms the size adaptation problem between the video to be played and the playback view into the size adaptation between the video frame and the texture object during texture loading. Different adaptive modes can correspond to different size adaptation logics for video frames and texture objects, greatly improving the freedom of size adaptation between the video to be played and the playback view. Compared with the size adaptation between video and view, the adaptation between image frames and texture objects is more efficient and has better real-time performance. Size adaptation can be achieved without adjusting the playback view during playback, without generating additional memory consumption. This improves the freedom of size adaptation between the video to be played and the playback view while ensuring the superior performance of the playback view.

[0156] In the process of drawing the data of the texture object onto the drawing surface object corresponding to the playback view, the texture coordinates of the points of the drawing surface object can be used to sample the texture image of the texture object to obtain the texture image of the drawing surface object. The texture image is then displayed on the display device through the playback view, realizing the video screen display, so that the user can see the video playback screen.

[0157] refer to Figure 6 and Figure 12 As shown, the drawing surface object serves as the display container for video playback, provided by the playback view. It acts as the rendering window for OpenGL. Thus, OpenGL and MediaCodec, combined with the Surface, form the video playback framework, resulting in a high-efficiency, highly flexible video playback component that adapts to video dimensions. This implements a height-adaptive video player for the Android platform. In this video playback component, MediaCodec handles decoding, while OpenGL and the Surface handle rendering to the screen, allowing users to see the video. Essentially, the playback view provides the basic environment for OpenGL operations, including creating and managing the OpenGL context.

[0158] Based on the above description, the overall operational logic of the video playback process includes: adding a playback view to the view tree to provide a drawing surface object; initializing the rendering window and creating a texture object through the drawing surface object in the OpenGL thread; decoding the video to be played using MediaCodec to obtain multiple video frames and continuously acquiring the decoded video frame data; uploading the video frame data of multiple video frames to the texture object according to the size matching information corresponding to the current adaptive mode; and continuously drawing the texture object data to the drawing window (i.e., the drawing surface object) in the OpenGL thread to display the video frame. This process of decoding, uploading video frame data to the texture object, and drawing texture object data to the drawing window is repeated until the video playback is complete. During the process of uploading video frame data to the texture object, the size adaptation between the video frame and the texture object is implemented. This size adaptation can be performed in real time and can also be adjusted in real time as needed.

[0159] The parameters of the playback view can be customized. The playback view is displayed on the screen and serves as the playback area. In full-screen mode, the size of the playback view can be the screen size; in non-full-screen mode, the size of the playback view is smaller than the screen size. This allows the size of the playback view to be determined based on the size of the playback area, and the size of the drawing surface objects and texture objects to be determined based on the size of the playback view. This enables the adaptation of the video frame size to the playback area through texture object and video frame size matching.

[0160] In this embodiment of the application, if the playback view is a texture view and the playback view is added to the view tree, then if a display transformation operation for the view tree is obtained, the display parameters of the playback view can be adjusted according to the display transformation operation. The display parameters include position parameters, size parameters, etc., which are used to perform transformation operations such as rotation, scaling, and translation on the playback view. That is, the playback view can be transformed, animated, or cropped together with other View interfaces to enrich the form of video playback.

[0161] In this embodiment of the application, if the playback view is a surface view, the operation of continuously drawing texture objects onto the drawing surface object corresponding to the playback view can be performed by a background thread other than the main thread, without occupying the main thread, reducing main thread lag and improving playback performance.

[0162] The process of determining size adaptation information under different adaptive modes is described below.

[0163] Size matching information includes the location information of the area to be displayed in multiple video frames, as well as the location information of the mapped area of ​​the texture object. The two have a mapping relationship. Based on the mapping relationship between the texture object and the drawing surface object, the area to be displayed can be mapped to the drawing surface object.

[0164] As one possible implementation, if the current adaptive mode includes full-screen display, it means that the entire area of ​​the texture object is occupied. Therefore, the position information of the mapped area can be determined based on the entire area of ​​the texture object. For example, the position information of the entire area of ​​the texture object can be used as the position information of the mapped area. Then, based on the size information of multiple video frames and the size information of the mapped area, the position information of the area to be displayed is determined, ensuring that the area to be displayed and the mapped area have the same aspect ratio. This guarantees proportional playback of video frames in full-screen display mode and ensures a good video display effect. It should be noted that the entire area of ​​the texture object is a region that has a mapping relationship with the entire visible area of ​​the playback view. Therefore, the occupation of the entire area of ​​the texture object means that the entire visible area of ​​the drawing surface object is occupied, which is equivalent to the occupation of the entire visible area of ​​the playback view.

[0165] In determining the location information of the area to be displayed, the location information of the first feature point in the area to be displayed can be determined based on the current adaptive mode; the dimensions of the first side and the second side adjacent to the first side of the video display area can be determined based on the size information of multiple video frames and the size information of the mapped area; and the location information of the area to be displayed can be determined based on the location information of the first feature point, the size of the first side, and the size of the second side. In this way, using the first feature point as an anchor point, the area to be displayed can be constructed based on the dimensions of the first and second sides, thereby determining the location information of the area to be displayed and improving the accuracy of location information determination.

[0166] Wherein, if the current adaptive mode includes displaying the first vertex side portion, the first feature point is the first vertex, and the position information of the first vertex is determined based on the position information of the first vertex in multiple video frames, the first vertex side may include the upper left vertex side, the lower left vertex side, the upper right vertex side, and the lower right vertex side; if the current adaptive mode includes displaying the center portion, the first feature point is the region center point, and the position information of the region center point is determined based on the position information of the center points in multiple video frames.

[0167] In determining the dimensions of the first and second sides of the area to be displayed, if the aspect ratio of the multiple video frames is determined to be equal to the aspect ratio of the mapped area based on the size information of the multiple video frames and the size information of the mapped area, then the height of the multiple video frames is used as the dimension of the first side of the area to be displayed, and the width of the multiple video frames is used as the dimension of the second side adjacent to the first side. In other words, the entire area of ​​each of the multiple video frames is used as the area to be displayed.

[0168] Let w be the width and height of the multiple video frames. v and h v Aspect R v =w v / hv The entire region of multiple video frames can be represented as (0,0)-(w v ,h v ); Let w be the width and height of the playback view. t and h t Aspect R t =w t / h t The playback view can be represented as (0,0)-(w t ,h t The width and height of the texture object corresponding to the playback view are denoted as w. s and h s Aspect R s =w s / h s The entire region of a texture object can be represented as (0,0)-(w s ,h s Once the mapping area of ​​the texture object is determined, it can be mapped to the video playback area of ​​the playback view. This allows the video to be displayed within the video playback area of ​​the playback view after the texture object loads a video frame. The aspect ratio of the playback view and the texture object can be equal, i.e., r... t =w t / h t =w s / h s =r s .

[0169] In a scene where the entire screen is filled, the entire area of ​​the texture object is used as the mapping area. The width and height of the mapping area are then the width and height of the texture object, respectively, which is w. s and h s The aspect ratio is r s The mapped region can be represented as (0,0)-(w s ,h s In the playback view, the video playback area corresponding to the mapped area can be represented as (0,0)-(w t ,h t ), which is the entire area of ​​the playback view.

[0170] For details, please refer to Figure 7 As shown, if the aspect ratio of multiple video frames is determined to be equal to the aspect ratio (r) of the mapped region, s =r v Then the height h of multiple video frames can be... v The size of the first side is the width w of multiple video frames. v The size of the second side determines the area to be displayed as the entire area of ​​each of the multiple video frames, represented as (0,0)-(w v ,hv The area to be displayed is (0,0)-(w). v ,h v Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0171] In determining the dimensions of the first and second sides of the area to be displayed, if, based on the size information of multiple video frames and the size information of the mapped area, the aspect ratio of the multiple video frames is determined to be greater than that of the mapped area, then the entire height of the multiple video frames can be displayed to retain more video content. The width is then cropped, meaning the height of the multiple video frames is used as the dimension of the first side of the area to be displayed. Then, based on the aspect ratio of the mapped area and the dimension of the first side, the dimension of the second side adjacent to the first side is determined. Since the aspect ratios of the mapped area and the area to be displayed are equal, it's equivalent to using the aspect ratio of the mapped area as the aspect ratio of the area to be displayed. Finally, based on the aspect ratio of the area to be displayed and the dimension of the first side, the dimension of the second side is determined. In this way, the largest area from multiple video frames is determined as the area to be displayed that meets the aspect ratio requirements, preserving the content of multiple video frames to the maximum extent.

[0172] Specifically, if it is determined that the aspect ratio of multiple video frames is greater than the aspect ratio of the mapped region (r... v >r s Then the height h of multiple video frames can be... v The dimension of the first side, i.e., the height of the area to be displayed, ranges from 0 to h. v The aspect ratio r of the mapped region s As the aspect ratio of the area to be displayed, the dimension of the second side can be denoted as h. v r s .

[0173] refer to Figure 8 As shown, if the current adaptive mode includes displaying the first vertex side portion, and the first vertex side portion is the top-left vertex side, then the position of the first vertex in multiple video frames is (0,0), which can be used as the first feature point in the area to be displayed, i.e., the position of the first vertex. Based on the position (0,0) of the first vertex in the area to be displayed, and the first side dimension h... v Second side dimension h v r s The area to be displayed can be determined as (0,0)-(r). s h v ,h v The area to be displayed is (0,0)-(r).s h v ,h v Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0174] refer to Figure 8 As shown, if the current adaptive mode includes displaying the first vertex side portion, and the first vertex side portion is the bottom right vertex side, then the position of the first vertex in multiple video frames is (w v ,h v This can be used as the position of the first feature point in the area to be displayed, i.e., the position of the first vertex. Based on the position of the first vertex of the area to be displayed (w... v ,h v ), and the first side dimension h v Second side dimension h v r s The area to be displayed can be determined as (w) v -h v r s ,0)-(w v ,h v The area to be displayed soon (w) v -h v r s ,0)-(w v ,h v Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0175] refer to Figure 8 As shown, if the current adaptive mode includes the display center portion, then the position of the first vertex of multiple video frames is the center position (w). v / 2,h v / 2), which can be used as the first feature point of the area to be displayed, i.e., the location of the center point of the area. Based on the location of the center of the area to be displayed (w v / 2,h v / 2), and the first side dimension h v Second side dimension h v r s The area to be displayed can be determined as (w) v / 2 - h v r s / 2,0)-(wv / 2 + h v r s / 2,h v The area to be displayed soon (w) v / 2-h v r s / 2,0)-(w v / 2 + h v r s / 2,h v Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0176] In determining the dimensions of the first and second sides of the area to be displayed, if, based on the dimensions of multiple video frames and the dimensions of the mapped area, the aspect ratio of the multiple video frames is determined to be smaller than that of the mapped area, then the width of the multiple video frames can be fully displayed to retain more video content. The width of these video frames is then cropped, and the width of the multiple video frames is used as the dimension of the second side of the area to be displayed. Then, based on the aspect ratio of the mapped area and the dimension of the second side, the dimension of the first side adjacent to the second side is determined. Since the aspect ratios of the mapped area and the area to be displayed are equal, it is equivalent to using the aspect ratio of the mapped area as the aspect ratio of the area to be displayed. The dimension of the first side is then determined based on the aspect ratio of the area to be displayed and the dimension of the second side. In this way, the largest area from multiple video frames is determined as the area to be displayed that meets the aspect ratio requirements, thus preserving the content of multiple video frames to the maximum extent.

[0177] Specifically, if it is determined that the aspect ratio of multiple video frames is less than the aspect ratio (r) of the mapped region... v <r s Then the width w of multiple video frames can be... v The dimension of the second side, i.e., the width of the area to be displayed, ranges from 0 to w. v The aspect ratio r of the mapped region s As the aspect ratio of the area to be displayed, the dimension of the first side can be denoted as w. v / r s .

[0178] refer to Figure 9As shown, if the current adaptive mode includes displaying the first vertex side portion, and the first vertex side portion is the top-left vertex side, then the position of the first vertex in multiple video frames is (0,0), which can be used as the first feature point in the area to be displayed, i.e., the position of the first vertex. Based on the position (0,0) of the first vertex in the area to be displayed, and the first side dimension w v / r s Second side dimension w v The area to be displayed can be determined to be (0,0)-(w v ,w v / r s The area to be displayed is (0,0)-(w). v ,w v / r s Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0179] refer to Figure 9 As shown, if the current adaptive mode includes displaying the first vertex side portion, and the first vertex side portion is the bottom right vertex side, then the position of the first vertex in multiple video frames is (w v ,h v This can be used as the position of the first feature point in the area to be displayed, i.e., the position of the first vertex. Based on the position of the first vertex of the area to be displayed (w... v ,h v ), and the first side dimension w v / r s Second side dimension w v The area to be displayed can be determined as (0, h). v -w v / r s )-(w v ,h v The area to be displayed (0, h) v -w v / r s )-(w v ,h v Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0180] refer to Figure 9As shown, if the current adaptive mode includes the display center portion, then the position of the first vertex of multiple video frames is the center position (w). v / 2,h v / 2), which can be used as the first feature point of the area to be displayed, i.e., the location of the center point of the area. Based on the location of the center of the area to be displayed (w v / 2,h v / 2), and the first side dimension w v / r s Second side dimension w v The area to be displayed can be determined as (0, h). v / 2 - w v / r s / 2)-(w v ,h v / 2 + w v / r s / 2). The area to be displayed (0, h) v / 2-w v / r s / 2)-(w v ,h v / 2 + w v / r s / 2) Mapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0181] As another possible implementation, if the current adaptive mode includes a full-view display, it means that it is necessary to prioritize displaying all content in the video frame. In this case, the position information of the area to be displayed is determined based on the total area of ​​multiple video frames. For example, the position information of the total area of ​​multiple video frames can be used as the position information of the area to be displayed. Then, based on the size information of the area to be displayed and the size information of the texture object, the position information of the mapping area is determined so that the area to be displayed and the mapping area have the same aspect ratio. This ensures that the video frames are played proportionally in the full-view display mode and guarantees a good display effect for the video.

[0182] In determining the location information of the mapping region, the location information of the second feature point in the mapping region can be determined based on the current adaptive mode; the dimensions of the third side and the fourth side adjacent to the third side of the mapping region can be determined based on the size information of the area to be displayed and the size information of the texture object; and the location information of the mapping region can be determined based on the location information of the second feature point, the size of the third side, and the size of the fourth side. In this way, using the second feature point as an anchor point, the mapping region can be constructed based on the size of the third and fourth sides, thereby determining the location information of the mapping region and improving the accuracy of location information determination.

[0183] Wherein, if the current adaptive mode includes display on the second vertex side, the second feature point is the second vertex, and the position information of the second vertex is determined according to the position information of the second vertex of the texture object, the second vertex side may include the top left vertex side, the bottom left vertex side, the top right vertex side, and the bottom right vertex side; if the current adaptive mode includes display in the center, then the second feature point is the center point of the region, and the position information of the center point of the region is determined according to the position information of the center point of the texture object.

[0184] In determining the third and fourth sides of the mapping area, if the aspect ratio of the area to be displayed is equal to that of the texture object, the width of the texture object is used as the dimension of the third side of the mapping area; the height of the texture object is used as the dimension of the fourth side adjacent to the third side. In other words, the entire area of ​​the mapping area is considered the mapping area.

[0185] In a scenario where the screen fills the entire display, the area to be displayed comprises the entire areas of multiple video frames. Therefore, the width and height of the area to be displayed are the width and height of the multiple video frames, respectively, which is w. v and h v The aspect ratio is r v The area to be displayed can be represented as (0,0)-(w v ,h v ).

[0186] For details, please refer to Figure 7 As shown, if the aspect ratio of the area to be displayed is equal to the aspect ratio of the texture object (r... s =r v Then the width w of the texture object can be set. s The size of the third side of the mapped region; the height h of the texture object. s As the dimension of the fourth side adjacent to the third side, the mapping region is the entire area of ​​the texture object, represented as (0,0)-(w s ,h s The area to be displayed is (0,0)-(w). v ,h vMapped to the mapping region (0,0)-(w s ,h s The corresponding video playback area in the playback view is (0,0)-(w). t ,h t ).

[0187] In determining the third and fourth sides of the mapping area, if the aspect ratio of the area to be displayed is greater than that of the texture object based on the dimensions of multiple video frames and the size information of the mapping area, then the width direction of the playback view can be filled to reserve a larger video playback area. In other words, the width of the texture object is used as the size of the third side of the mapping area. The size of the fourth side, adjacent to the third side, is then determined based on the aspect ratio of the area to be displayed and the size of the third side. Since the aspect ratios of the mapping area and the area to be displayed are equal, it's equivalent to using the aspect ratio of the area to be displayed as the aspect ratio of the mapping area. Then, the size of the fourth side is determined based on the aspect ratio of the mapping area and the size of the third side. In this way, the largest area from the texture object is determined as the mapping area that meets the aspect ratio requirement, maximizing the display area.

[0188] Specifically, if it is determined that the aspect ratio of multiple video frames is greater than the aspect ratio of the texture object (r... v >r s Then the width w of the texture object can be set. s The dimension of the third side of the mapping region, i.e., the width of the texture object, ranges from 0 to w. s The aspect ratio r of the area to be displayed v As the aspect ratio of the mapped region, the dimension of the fourth side can be denoted as w. s / r v .

[0189] refer to Figure 10 As shown, if the current adaptive mode includes displaying the second vertex side portion, and the second vertex side portion is the top-left vertex side, then the position of the second vertex of the texture object is (0,0), which can be used as the second feature point in the mapping region, i.e., the position of the second vertex. Based on the position (0,0) of the second vertex in the mapping region, and the third side dimension w... s and the fourth side dimension w s / r v The mapping region can be determined to be (0,0)-(w s ,w s / r v The area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (0,0)-(w s ,w s / r vThe corresponding video playback area in the playback view can be represented as (0,0)-(w t ,w s / r v ).

[0190] refer to Figure 10 As shown, if the current adaptive mode includes displaying the second vertex side portion, and the second vertex side portion is the bottom right vertex side, then the position of the second vertex of the texture object is (w s ,h s This can be used as the position of the second feature point in the mapped region, i.e., the position of the second vertex. Based on the position of the second vertex in the mapped region (w... s ,h s ), and the third side dimension w s and the fourth side dimension w s / r v The mapping region can be determined to be (0, h). s -w s / r v )-(w s ,h s The area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (0, h) s -w s / r v )-(w s ,h s The corresponding video playback area in the playback view can be represented as (0, h). t -w t / r v )-(w t ,h t ).

[0191] refer to Figure 10 As shown, if the current adaptive mode includes the display center portion, then the position of the second vertex of the texture object is the center position (w). s / 2,h s / 2), which can be used as the second feature point of the mapped region, i.e., the location of the region center. Based on the location of the region center (w) of the mapped region... s / 2,h s / 2), and the third side dimension w s and the fourth side dimension w s / r v The mapping region can be determined to be (0, h). s / 2-w s / r v / 2)-(w s ,h s / 2+ws / r v / 2), the area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (0, h) s / 2-w s / r v / 2)-(w s ,h s / 2+w s / r v / 2), the corresponding video playback area in the playback view can be represented as (0, h t / 2-w t / r v / 2)-(w t ,h t / 2+w t / r v / 2).

[0192] In determining the third and fourth sides of the mapping area, if the aspect ratio of the area to be displayed is determined to be smaller than that of the texture object based on the dimensions of multiple video frames and the size information of the mapping area, then the height direction of the playback view can be filled to preserve a larger video playback area. In other words, the height of the texture object is used as the size of the fourth side of the mapping area. The size of the third side adjacent to the fourth side is then determined based on the aspect ratio of the area to be displayed and the size of the fourth side. Since the aspect ratios of the mapping area and the area to be displayed are equal, it's equivalent to using the aspect ratio of the area to be displayed as the aspect ratio of the mapping area. Then, the size of the third side is determined based on the aspect ratio of the mapping area and the size of the fourth side. In this way, the largest area from the texture object is determined as the mapping area that meets the aspect ratio requirement, maximizing the display area.

[0193] Specifically, if the aspect ratio of the area to be displayed is greater than the aspect ratio of the texture object (r... v <r s Then the height h of the texture object can be set. s The dimension of the fourth side of the mapping region, i.e., the height of the texture object, ranges from 0 to h. s The aspect ratio r of the area to be displayed v As the aspect ratio of the mapped region, the dimension of the third side can be denoted as r. v h s .

[0194] refer to Figure 11As shown, if the current adaptive mode includes displaying the second vertex side portion, and the second vertex side portion is the top-left vertex side, then the position of the second vertex of the texture object is (0,0), which can be used as the second feature point in the mapping region, i.e., the position of the second vertex. Based on the position (0,0) of the second vertex in the mapping region, and the third side size r... v h s and the fourth side dimension h s The mapping region can be determined to be (0,0)-(r). v h s ,h s The area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (0,0)-(r) v h s ,h s The corresponding video playback area in the playback view can be represented as (0,0)-(r). v h t ,h t ).

[0195] refer to Figure 11 As shown, if the current adaptive mode includes displaying the second vertex side portion, and the second vertex side portion is the bottom right vertex side, then the position of the second vertex of the texture object is (w s ,h s This can be used as the position of the second feature point in the mapped region, i.e., the position of the second vertex. Based on the position of the second vertex in the mapped region (w... s ,h s ), and the third side dimension r v h s and the fourth side dimension h s The mapping region can be determined as (w) s -r v h s ,0)-(w s ,h s The area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (w) s -r v h s ,0)-(w s ,h s The corresponding video playback area in the playback view can be represented as (w t -r v h t ,0)-(w t ,h t ).

[0196] refer to Figure 11 As shown, if the current adaptive mode includes the display center portion, then the position of the second vertex of the texture object is the center position (w). s / 2,h s / 2), which can be used as the second feature point of the mapped region, i.e., the location of the region center. Based on the location of the region center (w) of the mapped region... s / 2,h s / 2), and the third side dimension r v h s and the fourth side dimension h s The mapping region can be determined as (w) s / 2-r v h s / 2,0)-(w s / 2+r v h s / 2,h s The area to be displayed is (0,0)-(w v ,h v Mapped to the mapping region (w) s / 2-r v h s / 2,0)-(w s / 2+r v h s / 2,h s The corresponding video playback area in the playback view can be represented as (w t / 2-r v h t / 2,0)-(w t / 2+r v h t / 2,h t ).

[0197] Based on the video playback method provided in the embodiments of this application, the embodiments of this application also provide a video playback device, see reference. Figure 13 The diagram shown is a structural block diagram of a video playback device provided in an embodiment of this application. The video playback device 1300 includes:

[0198] The video frame acquisition unit 1301 is used to obtain multiple video frames based on the video to be played.

[0199] The size adaptation information determination unit 1302 is used to determine the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object;

[0200] The texture loading unit 1303 is used to load the video frame data of the multiple video frames into the texture object according to the playback order of the multiple video frames based on the size adaptation information, and the display areas of the multiple video frames are respectively mapped to the mapping area;

[0201] The texture drawing unit 1304 is used to continuously draw the data of the texture object onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

[0202] Optionally, the size adaptation information determination unit includes:

[0203] The first position determination unit is used to determine the position information of the mapped region based on the entire region of the texture object if the current adaptive mode includes full-screen display.

[0204] The second position determination unit is used to determine the position information of the area to be displayed based on the size information of the plurality of video frames and the size information of the mapping area, so that the area to be displayed and the mapping area have the same aspect ratio.

[0205] Optionally, the second position determining unit includes:

[0206] The first feature point position determination unit is used to determine the position information of the first feature point in the area to be displayed according to the current adaptive mode; if the current adaptive mode includes displaying the first vertex side portion, the first feature point is the first vertex, and the position information of the first vertex is determined according to the position information of the first vertices of the multiple video frames; if the current adaptive mode includes displaying the center portion, the first feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center points of the multiple video frames.

[0207] The first size determination unit is used to determine the size of the first side of the area to be displayed and the size of the second side adjacent to the first side based on the size information of the plurality of video frames and the size information of the mapping area;

[0208] The display area determination subunit is used to determine the position information of the display area based on the position information of the first feature point, the size of the first side and the size of the second side.

[0209] Optionally, the first size determination unit is specifically used for:

[0210] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is greater than the aspect ratio of the mapping area, the height of the multiple video frames is used as the size of the first side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the first side, the size of the second side adjacent to the first side is determined.

[0211] If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is less than the aspect ratio of the mapping area, the width of the multiple video frames is used as the size of the second side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the second side, the size of the first side adjacent to the second side is determined.

[0212] Optionally, if the current adaptive mode fills the entire screen, the texture loading unit includes:

[0213] An adaptive module is used to extract the image to be displayed by sequentially capturing the multiple video frames according to the playback order of the multiple video frames, based on the location information of the area to be displayed.

[0214] The texture loading subunit is used to sequentially load the image data of the image to be displayed into the texture object.

[0215] Optionally, the size adaptation information determination unit includes:

[0216] The third position determination unit is used to determine the position information of the area to be displayed based on the entire area of ​​the plurality of video frames if the current adaptive mode includes a full view display.

[0217] The fourth position determination unit is used to determine the position information of the mapping area based on the size information of the area to be displayed and the size information of the texture object, so that the area to be displayed and the mapping area have the same aspect ratio.

[0218] Optionally, the fourth position determination unit includes:

[0219] The second feature point position determination unit is used to determine the position information of the second feature point in the mapping area according to the current adaptive mode; if the current adaptive mode includes second vertex side display, the second feature point is the second vertex, and the position information of the second vertex is determined according to the position information of the second vertex of the texture object; if the current adaptive mode includes center display, the second feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center point of the texture object.

[0220] The second size determination unit is used to determine the size of the third side of the mapping area and the size of the fourth side adjacent to the third side based on the size information of the area to be displayed and the size information of the texture object.

[0221] The mapping region determination subunit is used to determine the location information of the mapping region based on the location information of the second feature point, the size of the third side, and the size of the fourth side.

[0222] Optionally, the second size determination unit is specifically used for:

[0223] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is greater than the aspect ratio of the texture object, the width of the texture object is used as the size of the third side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the third side, the size of the fourth side adjacent to the third side is determined.

[0224] If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is less than the aspect ratio of the texture object, the height of the texture object is used as the size of the fourth side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the fourth side, the size of the third side adjacent to the fourth side is determined.

[0225] Optionally, the device further includes:

[0226] A playback view adding unit is used to add the playback view to the view tree to obtain the drawing surface object corresponding to the playback view; the size information of the drawing surface object is determined according to the size information of the playback view;

[0227] The texture object creation unit is used to create a texture object based on the drawing surface object corresponding to the playback view; the size information of the texture object is determined based on the size information of the drawing surface object.

[0228] Optionally, the playback view is a texture view, and the device further includes:

[0229] The display parameter adjustment unit is used to adjust the display parameters of the playback view according to the display transformation operation if a display transformation operation for the view tree is obtained.

[0230] Optionally, the device further includes:

[0231] An adaptive mode switching unit is configured to, during video playback using the playback view, update the current adaptive mode according to a target mode in response to a mode switching operation including a target mode, wherein the target mode is one of the plurality of adaptive modes.

[0232] As can be seen from the above technical solution, multiple video frames can be obtained based on the video to be played. Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the playback view, the size adaptation information under the current adaptive mode can be determined. The size information of the texture object is determined according to the size of the playback view corresponding to the texture object. The current adaptive mode is one of several adaptive modes. The size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object corresponding to the playback view. The areas to be displayed in the multiple video frames are used to map to the mapped areas respectively. It can be understood that when the area to be displayed is smaller than the overall area of ​​the video frame, the mapping process is equivalent to truncating multiple video frames; when the mapped area is smaller than the overall area of ​​the texture object, the mapping process is equivalent to selecting a region of the texture object. Therefore, through the size adaptation information under the current adaptive mode, the region selection logic of the video frame and the texture object under the current adaptive mode can be defined to achieve size adaptation between the video frame and the texture object under the current adaptive mode.

[0233] Then, based on the size adaptation information, the video frame data of multiple video frames can be loaded into the texture object in sequence according to the playback order of multiple video frames, and the data of the texture object can be continuously drawn onto the drawing surface object corresponding to the playback view so as to use the playback view for video playback. In other words, based on the region selection logic defined by the size adaptation information in the current adaptive mode, texture loading and texture object drawing of video frames can be performed. That is, by adapting the size of video frames and texture objects, size adaptation between video frames and the playback view is achieved, so that the area to be displayed in the video frame can be displayed in the video playback area corresponding to the mapped area in the playback view. In this way, the size adaptation problem between the video to be played and the playback view is transformed into the size adaptation between video frames and texture objects during texture loading. Different adaptive modes can correspond to different size adaptation logics for video frames and texture objects, greatly improving the freedom of size adaptation between the video to be played and the playback view. Compared with the size adaptation between the video to be played and the playback view, the adaptation between image frames and texture objects is more efficient and has better real-time performance. Size adaptation can be achieved without adjusting the playback view during playback, without generating additional memory consumption. In this way, while improving the freedom of size adaptation between the video to be played and the playback view, the superior performance of the playback view is guaranteed.

[0234] This application also provides a computer device, which is the computer device described above, and may include a terminal device or a server. The aforementioned video playback device may be configured in this computer device. The computer device will now be described in conjunction with the accompanying drawings.

[0235] If the computer device is a terminal device, please refer to Figure 14 As shown, this application provides a terminal device, taking a mobile phone as an example:

[0236] Figure 14 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 14 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 14 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0237] The following is combined with Figure 14 A detailed introduction to each component of a mobile phone:

[0238] The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0239] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0240] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0241] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0242] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0243] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0244] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0245] The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0246] The mobile phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0247] In this embodiment, the processor 1480 included in the terminal device also has the following functions:

[0248] Multiple video frames are obtained based on the video to be played;

[0249] Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object, the size adaptation information under the current adaptive mode is determined; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object;

[0250] Based on the size adaptation information, according to the playback order of the multiple video frames, the video frame data of the multiple video frames are loaded into the texture object in sequence, and the display areas of the multiple video frames are respectively mapped to the mapping area;

[0251] The data of the texture object is continuously drawn onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

[0252] If the computer device is a server, this application embodiment also provides a server; please refer to [link to relevant documentation]. Figure 15 As shown, Figure 15 This is a structural diagram of a server 1500 provided in an embodiment of this application. The server 1500 can vary significantly due to different configurations or performance. It may include one or more processors 1522, such as a Central Processing Unit (CPU), a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the processor 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

[0253] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server. TM Mac OS X TM UnixTM Linux TM FreeBSD TM etc.

[0254] The steps performed by the server in the above embodiments can be based on Figure 15 The server structure shown.

[0255] In addition, this application also provides a computer-readable storage medium for storing a computer program for executing the methods provided in the above embodiments.

[0256] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0257] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to computer programs. The aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it performs the steps of the above method embodiments. The aforementioned computer-readable storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disk, etc., and other media that can store computer programs.

[0258] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0259] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video playback method, characterized in that, The method includes: Multiple video frames are obtained based on the video to be played; Based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object, the size adaptation information under the current adaptive mode is determined; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object; Based on the size adaptation information, according to the playback order of the multiple video frames, the video frame data of the multiple video frames are loaded into the texture object in sequence, and the display areas of the multiple video frames are respectively mapped to the mapping area; The data of the texture object is continuously drawn onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

2. The method according to claim 1, characterized in that, The step of determining the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object includes: If the current adaptive mode includes full-screen display, then the position information of the mapped region is determined based on the entire area of ​​the texture object; Based on the size information of the multiple video frames and the size information of the mapped area, the position information of the area to be displayed is determined, so that the area to be displayed and the mapped area have the same aspect ratio.

3. The method according to claim 2, characterized in that, Determining the location information of the area to be displayed based on the size information of the multiple video frames and the size information of the mapped area includes: According to the current adaptive mode, the position information of the first feature point in the area to be displayed is determined; if the current adaptive mode includes displaying the first vertex side portion, the first feature point is the first vertex, and the position information of the first vertex is determined according to the position information of the first vertices of the multiple video frames; if the current adaptive mode includes displaying the center portion, the first feature point is the center point of the area, and the position information of the center point of the area is determined according to the position information of the center points of the multiple video frames. Based on the size information of the multiple video frames and the size information of the mapping area, determine the size of the first side of the area to be displayed and the size of the second side adjacent to the first side; The location information of the area to be displayed is determined based on the location information of the first feature point, the size of the first side, and the size of the second side.

4. The method according to claim 3, characterized in that, The step of determining the size of the first side of the area to be displayed and the size of the second side adjacent to the first side based on the size information of the plurality of video frames and the size information of the mapped area includes: If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is greater than the aspect ratio of the mapping area, the height of the multiple video frames is used as the size of the first side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the first side, the size of the second side adjacent to the first side is determined. If, based on the size information of the multiple video frames and the size information of the mapping area, it is determined that the aspect ratio of the multiple video frames is less than the aspect ratio of the mapping area, the width of the multiple video frames is used as the size of the second side of the area to be displayed; and based on the aspect ratio of the mapping area and the size of the second side, the size of the first side adjacent to the second side is determined.

5. The method according to claim 2, characterized in that, If the current adaptive mode fills the entire screen, the step of loading the video frame data of the multiple video frames into the texture object sequentially according to the playback order of the multiple video frames, based on the size adaptation information, includes: Based on the location information of the area to be displayed, the multiple video frames are sequentially extracted according to their playback order to obtain the image to be displayed. The image data of the image to be displayed is loaded into the texture object in sequence.

6. The method according to claim 1, characterized in that, The step of determining the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object includes: If the current adaptive mode includes a full content overview display, then the location information of the area to be displayed is determined based on the entire area of ​​the multiple video frames; Based on the size information of the area to be displayed and the size information of the texture object, the position information of the mapping area is determined so that the area to be displayed and the mapping area have the same aspect ratio.

7. The method according to claim 6, characterized in that, The step of determining the position information of the mapped region based on the size information of the area to be displayed and the size information of the texture object includes: Based on the current adaptive mode, the position information of the second feature point in the mapping region is determined; if the current adaptive mode includes side display of the second vertex, the second feature point is the second vertex, and the position information of the second vertex is determined according to the position information of the second vertex of the texture object; if the current adaptive mode includes center display, the second feature point is the center point of the region, and the position information of the center point of the region is determined according to the position information of the center point of the texture object. Based on the size information of the area to be displayed and the size information of the texture object, determine the size of the third side of the mapping area and the size of the fourth side adjacent to the third side; The location information of the mapping region is determined based on the location information of the second feature point, the size of the third side, and the size of the fourth side.

8. The method according to claim 7, characterized in that, The step of determining the size of the third side of the mapped area and the size of the fourth side adjacent to the third side based on the size information of the area to be displayed and the size information of the texture object includes: If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is greater than the aspect ratio of the texture object, the width of the texture object is used as the size of the third side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the third side, the size of the fourth side adjacent to the third side is determined. If, based on the size information of the area to be displayed and the size information of the texture object, it is determined that the aspect ratio of the area to be displayed is less than the aspect ratio of the texture object, the height of the texture object is used as the size of the fourth side of the mapped area; and based on the aspect ratio of the area to be displayed and the size of the fourth side, the size of the third side adjacent to the fourth side is determined.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Add the playback view to the view tree to obtain the drawing surface object corresponding to the playback view; the size information of the drawing surface object is determined according to the size information of the playback view. A texture object is created based on the drawing surface object corresponding to the playback view; the size information of the texture object is determined based on the size information of the drawing surface object.

10. The method according to claim 9, characterized in that, The playback view is a texture view, and the method further includes: If a display transformation operation is obtained for the view tree, the display parameters of the playback view are adjusted according to the display transformation operation.

11. The method according to any one of claims 1-8, characterized in that, The method further includes: During video playback using the playback view, in response to a mode switching operation corresponding to the target mode, the current adaptive mode is updated according to the target mode, where the target mode is one of the multiple adaptive modes.

12. A video playback device, characterized in that, The device includes: The video frame acquisition unit is used to obtain multiple video frames based on the video to be played. The size adaptation information determination unit is used to determine the size adaptation information under the current adaptive mode based on the current adaptive mode, the size information of the multiple video frames, and the size information of the texture object; the size information of the texture object is determined according to the size information of the playback view corresponding to the texture object; the current adaptive mode is one of multiple adaptive modes; the size adaptation information includes the position information of the area to be displayed in the multiple video frames and the position information of the mapped area in the texture object; The texture loading unit is used to load the video frame data of the multiple video frames into the texture object sequentially according to the playback order of the multiple video frames based on the size adaptation information, and the display areas of the multiple video frames are respectively mapped to the mapping area; The texture drawing unit is used to continuously draw the data of the texture object onto the drawing surface object corresponding to the playback view, so as to use the playback view for video playback.

13. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the video playback method according to any one of claims 1-11 according to instructions in the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which is loaded and executed by a processor to implement the video playback method as described in any one of claims 1-11.

15. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the video playback method according to any one of claims 1-11.

Images