Display device and video playing processing method

Abstract

This application discloses a display device and a video playback processing method. The display device constrains the width and height of a video container according to a preset aspect ratio. In response to the video container exceeding the screen area, the overlapping area between the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system. Then, based on the preset aspect ratio constraint, the original video aspect ratio is maintained, and the visible area is transformed to the video coordinate system according to a preset processing strategy to obtain the target rendering area. The video image of the target rendering area is rendered through the video container. Based on the visible area and the target rendering area, a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed to respond to touch events targeting the target rendering area according to the bidirectional mapping relationship. This application can effectively maintain the video aspect ratio, avoid problems such as screen aspect ratio distortion, detail distortion, and touch failure when the video exceeds the screen, and optimize the interactive experience of video playback.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a video playback processing method. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart touch TVs as an example, smart touch TVs are television products based on Internet application technology, equipped with open operating systems and chips, and possessing open application platforms. They can realize two-way human-computer touch interaction functions and integrate multiple functions such as audio-visual, entertainment, and data to meet the diverse and personalized needs of users.

[0003] When playing video, the video playback component of the display device uses a video container to hold the video and renders and displays it. During video playback, users can move the position of the video container using touch controls. When the video container is about to exceed the screen area of ​​the display device (i.e., out-of-screen state), the display device will reduce the size of the video container to ensure that it is fully displayed on the screen. At the same time, after reducing the size of the video container, the display device will also perform irregular pixel compression on the video image, forcibly adapting the complete video image to the adjusted container area.

[0004] However, the video display and touch response in the above-mentioned over-screen state not only cause the proportion of the corresponding screen content in the video to be out of proportion and the details to be distorted, affecting the display effect of the video, but also cause the correspondence between the touch coordinates and the original video screen content to be disordered due to the irregular compression of pixels. This results in the touch operation being unable to match the interactive area of ​​the screen content, causing touch failure and thus affecting the user's experience of using the display device. Summary of the Invention

[0005] This application provides a display device and a video playback processing method to solve the problems of decreased video display effect and touch failure during video playback in the display device.

[0006] In a first aspect, this application provides a display device, comprising: The display is configured to display screen content rendered by a video container in a screen area according to a screen coordinate system, the video container being used to render the video frame of the original video; the display is connected to a touch interaction module, the touch interaction module being used to receive touch events input by the user, the touch events including touch coordinates in the touch coordinate system; A controller is coupled to the display and the touch interaction module; the controller is configured to constrain the width and height dimensions of the video container according to a preset aspect ratio, and is further configured to: In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system; Based on the preset aspect ratio constraint, the aspect ratio of the original video is maintained, and the visible area in the container coordinate system is transformed to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area. The video image of the target rendering area is rendered through the video container; Based on the visible area and the target rendering area, a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed to respond to touch events targeting the target rendering area according to the bidirectional mapping relationship.

[0007] The above technical solution has the following beneficial effects or advantages: The display device constantly constrains the size of the video container by pre-setting the aspect ratio, avoiding video stretching and compression caused by irregular container adjustments. By precisely mapping the visible area and calculating the target rendering area, it ensures that only the effective video image is rendered when the video container exceeds the screen, preventing image aspect ratio distortion and detail loss. At the same time, it establishes a two-way mapping relationship between the screen and the touch coordinate system, enabling the touch coordinates to accurately match the interactive area of ​​the video image, solving the touch failure problem when the video exceeds the screen, and optimizing the visual effect of video playback and the touch interaction experience.

[0008] In some embodiments of this application, the controller performs constraint on the width and height dimensions of the video container according to a preset aspect ratio, specifically configured as follows: Obtain the initial width and initial height of the video container; Based on the initial width, the corresponding container adaptation height is calculated according to the preset aspect ratio; If the container adaptation height is less than or equal to the initial height, the container width of the video container is set to the initial width, and the container height of the video container is set to the container adaptation height. If the container adaptation height is greater than the initial height, the corresponding container adaptation width is calculated based on the initial height and the preset aspect ratio; then the container width of the video container is set to the container adaptation width, and the container height of the video container is set to the initial height.

[0009] The above technical solution has the following beneficial effects or advantages: By using the initial width and height as a basis for adaptation, the final container size that conforms to the preset aspect ratio is calculated and determined according to the initial size of the video container. This avoids aspect ratio imbalance caused by arbitrary size adjustments of the video container and ensures that the video image is always rendered based on a fixed ratio.

[0010] In some embodiments of this application, the controller performs the mapping of the overlapping area between the video container and the screen area to the container coordinate system of the video container, specifically configured as follows: Detect the left and top boundaries of the video container in the screen coordinate system; A container coordinate system is established with the top left corner of the video container as the origin, the direction to the right along the left boundary of the container as the positive direction of the horizontal coordinate, and the direction downward along the top boundary of the container as the positive direction of the vertical coordinate. In the container coordinate system, the maximum value between 0 and the negative left boundary of the container is determined as the visible left boundary of the visible area, the maximum value between 0 and the negative top boundary of the container is determined as the visible top boundary of the visible area, the minimum value between the container width and a first difference is determined as the visible right boundary of the visible area, and the minimum value between the container height and a second difference is determined as the visible bottom boundary of the visible area; wherein, the first difference is the difference between the screen width of the screen area and the left boundary of the container, and the second difference is the difference between the screen height of the screen area and the top boundary of the container.

[0011] The above technical solution has the following beneficial effects or advantages: By establishing a container coordinate system and combining it with the container boundary in the screen coordinate system, the four-way boundary of the visible area can be calculated. This allows for the accurate acquisition of the actual visible area of ​​the video container on the screen after it exceeds the screen size, providing an accurate and reliable data foundation for the subsequent calculation of the target rendering area.

[0012] In some embodiments of this application, the controller performs a preset processing strategy to transform the visible region in the container coordinate system to the video coordinate system of the original video, specifically configured as follows: Read the original video height and original video width of the original video, and read the container height and container width of the video container; Calculate the height ratio of the original video height to the container height, and calculate the width ratio of the original video width to the container width; In the video coordinate system, the product of the visible left boundary corresponding to the visible area and the width ratio is determined as the visible video left boundary, the product of the visible upper boundary corresponding to the visible area and the height ratio is determined as the visible video upper boundary, the product of the visible width corresponding to the visible area and the width ratio is determined as the visible video width, and the product of the visible height corresponding to the visible area and the height ratio is determined as the visible video height. A cropping window is generated based on the left boundary of the visible video, the top boundary of the visible video, the width of the visible video, and the height of the visible video; The original video is cropped using the cropping window to obtain the target rendering area in the video coordinate system.

[0013] The above technical solution has the following beneficial effects or advantages: By calculating the size ratio of the original video to the video container, the visible area ratio in the container coordinate system is converted to the video coordinate system. Then, the original video is cropped through the cropping window. This avoids irregular pixel compression of the video image and can extract the effective rendering image while maintaining the original aspect ratio of the video, thus preventing the problem of video detail distortion.

[0014] In some embodiments of this application, the controller generates a cropping window based on the left boundary of the visible video, the top boundary of the visible video, the width of the visible video, and the height of the visible video, specifically configured as follows: In the video coordinate system, the maximum value between 0 and the first reference value is determined as the left cropping boundary, the maximum value between 0 and the second reference value is determined as the top cropping boundary, the minimum value between 0 and the third reference value is determined as the cropping width, and the maximum value between 0 and the fourth reference value is determined as the cropping height; wherein, the first reference value is the minimum value between the original video width and the visible video left boundary, the second reference value is the minimum value between the original video height and the visible video top boundary, the third reference value is the minimum value between the third difference and the visible video width, the third difference is the difference between the original video width and the cropping left boundary, the fourth reference value is the minimum value between the fourth difference and the visible video height, and the fourth difference is the difference between the original video height and the cropping top boundary; A rectangular object is created based on the left cropping boundary, the top cropping boundary, the cropping width, and the cropping height to serve as the cropping window.

[0015] The above technical solution has the following beneficial effects or advantages: By determining the boundaries of the cropping window through multiple rounds of judgment, the cropping protection effect is achieved, which can prevent the cropping area from exceeding the actual size range of the original video and ensure that the cropping window is always within the effective area of ​​the original video. At the same time, using a rectangular object as the cropping window adapts to the display form of the video screen, ensuring that the target rendering area after cropping is a complete and effective image, further guaranteeing the integrity of the target rendering area.

[0016] In some embodiments of this application, the controller performs a preset processing strategy to transform the visible region in the container coordinate system to the video coordinate system of the original video, specifically configured as follows: Read the original video height and original video width of the original video, and read the visible area width and visible area height of the visible area; Calculate the ratio of the original video width to the original video height to obtain the video aspect ratio; Based on the height of the visible area, the corresponding video adaptation width is calculated according to the video aspect ratio; If the video adaptation width is less than or equal to the visible area width, the target rendering area of ​​the original video is determined according to the visible area height and the video adaptation width, based on the principle of centering display. If the video adaptation width is greater than the visible area width, the corresponding video adaptation height is calculated based on the visible area width and the video aspect ratio. The original video is then scaled according to the ratio of the visible area width to the video adaptation height, and the target rendering area of ​​the scaled original video is determined according to the principle of center display.

[0017] The above technical solution has the following beneficial effects or advantages: With the video aspect ratio as the core constraint, adaptive calculations are performed based on the size of the visible area. When the size exceeds the calculation range, the video is scaled proportionally to avoid disproportion of the video image. At the same time, the target rendering area is determined in conjunction with the principle of centering the display, so that the video image is presented in the visible area in a way that is more in line with the user's visual habits and improves the visual display effect of the video.

[0018] In some embodiments of this application, the controller determines the target rendering area of ​​the scaled original video according to the principle of center display, specifically configured as follows: Under the video coordinate system, read the target video width of the original video after scaling; Calculate half the difference between the width of the visible area and the width of the target video; In the video coordinate system, the half value is set as the left boundary of the target rendering area, and the upper boundary of the target rendering area is set to 0.

[0019] The above technical solution has the following beneficial effects or advantages: The left boundary is determined by calculating half the difference between the visible area and the target video width, while the upper boundary is set to 0. This allows for accurate positioning of the scaled video within the visible area, ensuring that the video image is horizontally centered within the visible area and guaranteeing the accuracy of the image rendering position.

[0020] In some embodiments of this application, the controller executes the rendering of video footage of the target rendering area through the video container, specifically configured as follows: The left boundary layout parameter of the video container is updated to the sum of the visible left boundary of the visible area and the left boundary of the target rendering area; the upper boundary layout parameter of the video container is updated to the sum of the visible upper boundary of the visible area and the upper boundary of the target rendering area. The right boundary layout parameter of the video container is updated to the sum of the left boundary layout parameter and the target video width, and the lower boundary layout parameter of the video container is updated to the sum of the upper boundary layout parameter and the target video height of the original video after scaling. The video image of the target rendering area is rendered using the video container according to the left boundary layout parameters, the upper boundary layout parameters, the right boundary layout parameters, and the lower boundary layout parameters.

[0021] The above technical solution has the following beneficial effects or advantages: By updating the layout parameters of the video container by aligning the visible area boundary with the target rendering area boundary, the display boundary of the video container matches the image boundary of the target rendering area. This ensures that the scaled and centered video image is accurately rendered and displayed within the visible area of ​​the screen, avoiding issues such as image offset and omissions.

[0022] In some embodiments of this application, the controller is further configured to: Record the associated parameters of the visible area, the target rendering area, and the bidirectional mapping relationship; In response to the video container's display area exceeding the screen area, if the difference between the currently visible area and the recorded visible area is less than a preset threshold, the associated parameters are reused. Render the video image of the target rendering area according to the associated parameters, and respond to touch events for the target rendering area according to the associated parameters.

[0023] The above technical solution has the following beneficial effects or advantages: By recording the associated parameters of the overscreen processing and setting reuse conditions, the calculated parameters can be reused directly when the overscreen state of the video container changes little. This eliminates the need to repeatedly perform complex calculations such as visible area mapping, target rendering area calculation, and bidirectional mapping relationship construction, thereby improving the response speed of the display device, reducing the device's resource consumption, and optimizing the display device's operating performance.

[0024] Secondly, this application also provides a video playback processing method applied to a display device. The display device's monitor is configured to display screen content rendered by a video container in a screen area according to a screen coordinate system. The video container is used to render the video frame of the original video. The monitor is connected to a touch interaction module, which is used to receive touch events input by a user. The touch events include touch coordinates in a touch coordinate system. The display device constrains the width and height dimensions of the video container according to a preset aspect ratio. The method includes: In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system; Based on the preset aspect ratio constraint, the video aspect ratio of the original video is maintained, and the visible area in the container coordinate system is transformed to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area. The video image of the target rendering area is rendered through the video container; Based on the visible area and the target rendering area, a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed to respond to touch events targeting the target rendering area according to the bidirectional mapping relationship.

[0025] The above technical solution has the following beneficial effects or advantages: The method maintains the original aspect ratio of the video, ensuring video display quality. Simultaneously, by constructing a bidirectional coordinate system mapping relationship, it allows touch operations to accurately match the interactive area of ​​the video screen, resolving touch failure issues caused by screen overload and optimizing the overall interactive experience of video playback on display devices. This method is widely applicable to display devices with touch functionality and is highly practical. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1This is a schematic diagram illustrating the operation scenarios of a display device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 These are schematic diagrams illustrating scenarios of screen distortion and touch failure in over-screen mode, provided in some embodiments of this application. Figure 5 A flowchart illustrating the process of constraining the width and height dimensions of a video container provided in some embodiments of this application; Figure 6 A flowchart illustrating a video playback processing method provided in some embodiments of this application; Figure 7 A schematic diagram illustrating the principle of the container coordinate system provided in some embodiments of this application; Figure 8 A schematic diagram illustrating the range of the visible area provided for some embodiments of this application; Figure 9 A schematic diagram illustrating the principle of the target rendering area under the clipping method provided in some embodiments of this application; Figure 10 This is a schematic diagram illustrating the principle of the target rendering area under the scaling method provided in some embodiments of this application; Figure 11 Interactive diagrams for video playback processing methods provided in some embodiments of this application; Figure 12 This is a schematic diagram illustrating the display effect of the cropping method provided in some embodiments of this application; Figure 13 This is a schematic diagram illustrating the display effect under scaling methods provided in some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0030] Figure 1This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0031] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0032] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0033] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0034] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0035] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0036] In some embodiments, detector 230 is used to collect signals from the external environment or signals interacting with the outside world. For example, detector 230 may include millimeter-wave radar, which can be used to detect whether a user is present within a preset range. Detector 230 may also include a voice acquisition unit to collect voice commands input by the user.

[0037] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0038] In some embodiments, the display 260 may also support touch interaction, meaning that a touch interaction module may be provided on the display device 200. This touch interaction module can be a built-in module of the display 260 or an external module connected to the display device 200 / display 260. For example, to implement touch interaction, the display 260 of the display device 200 may be equipped with a touch panel (TP). The touch panel can receive touch signals from fingers, touchpads, styluses, etc., and execute touch interaction responses according to a pre-set interaction strategy. Of course, the display device 200 may also be an integrated touchscreen.

[0039] For the display device 200 with an external touch interaction module, terminals, devices, and components with touch functionality can be connected via the device interface 240 of the display device 200. For example, the USB or USB-C interface of the display device 200 can be connected to a touchpad. The touchpad can receive touch data input by the user in real time and transmit the touch data to the display device 200 via the USB channel. The display device 200 can then perform interactive responses based on the received touch data and in conjunction with touch applications, enabling the display device 200 to support touch operations.

[0040] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0041] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0042] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0043] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0044] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0045] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0046] In some embodiments, the user input interface 280 can be used to receive instructions input by a user. The user input interface 280 may include at least one of a microphone, touchpad, sensor, remote control, etc. The display device 200 can then receive user-input instructions based on the user input interface 280 to perform interactive functions with the user.

[0047] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0048] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0049] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0050] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may run at least one application, which may be a Windows program included with the operating system, a system settings program, or a clock program, or an application developed by a third-party developer. In specific implementations, the applications in the application layer are not limited to the examples above.

[0051] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0052] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0053] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0054] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0055] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0056] The display device 200 can receive touch signals input by the user through the touch interaction module, according to the touch coordinate system. The touch interaction module converts the collected touch signals into coordinate data in the touch coordinate system, including touch position, touch pressure, touch duration, and gesture trajectory in multi-touch, to form corresponding touch events. For example, when a user's finger or stylus touches the surface of the touchscreen, the touch sensor array of the touchscreen detects changes in capacitance or resistance. The touch chip of the touchscreen performs analog-to-digital conversion and noise filtering on these raw signals to generate standardized touch data.

[0057] In some embodiments, the touch coordinate system can be a two-dimensional Cartesian coordinate system consistent with the screen coordinate system, with the origin set at the upper left or lower left corner of the screen area presented by the display 260. The horizontal axis represents the pixel position in the horizontal direction, and the vertical axis represents the pixel position in the vertical direction. The touch interaction module maps the collected physical coordinates to logical coordinates that match the current display resolution to ensure accurate correspondence between the touch position and the displayed content.

[0058] Touch events can include various gesture operations such as single click, double click, long press, swipe, pinch to zoom, and rotation. The touch interaction module identifies the specific gesture type based on the changing patterns and timing characteristics of the touch coordinates and assigns a unique event identifier code to each gesture event. For example, when the distance between two touch points gradually decreases, the touch interaction module recognizes it as a pinch gesture and generates a corresponding zoom control event; when a single touch point continuously moves beyond a preset distance threshold, it is recognized as a swipe gesture, and the vector data of the movement trajectory is recorded.

[0059] The generated touch events are encapsulated into standardized event structures and passed to the framework layer manager via the system event bus. The manager distributes the touch events to the corresponding applications for response processing based on the registration status of the currently foreground application window. Simultaneously, the touch interaction module also supports priority scheduling of touch events and anti-mistouch mechanisms. When abnormal touch patterns such as large-area palm contact are detected, invalid touch data can be automatically filtered, improving the accuracy of touch interaction and user experience.

[0060] Applications can play videos using a video playback component. This component uses a video container with a fixed aspect ratio to hold the video and renders the video image through the container. In some implementations, the aspect ratio of the video container is set according to the aspect ratio of the video, such as 16:9, 4:3, 21:9, etc., to ensure that the video image is not stretched or compressed during display.

[0061] In some embodiments, a video container is a dedicated display window object created in the system by a video playback component (such as a player app, a screen mirroring app, or a picture-in-picture app) at runtime by calling the view system and window manager interface of the framework layer.

[0062] In some embodiments, the video container, as a dedicated display window object, has independent layout parameters (position, width, height, and hierarchy in the screen coordinate system or the container coordinate system of the video container itself). Its position and display range can be flexibly adjusted by the video playback component or the user (through touch dragging) without affecting other displayed content on the screen. That is, the video container supports adjusting its display position and range in the screen coordinate system. It serves as a software intermediary layer connecting video decoding data and the hardware display, providing not only a flexibly adjustable rendering medium for the video image but also a dedicated response area for touch interaction.

[0063] In some embodiments, during video playback, the video playback component continuously parses the video stream data, decodes the compressed and encoded video frames in the stream data into raw image data, and renders them frame by frame into the video container according to the refresh sequence of the display device 200. The rendering engine supports hardware-accelerated decoding, which reduces the computational load on the central processing unit by calling the decoding pipeline of the graphics processing unit, thereby improving the smoothness of high-resolution video playback.

[0064] In some embodiments, the video playback component provides a complete playback control interface, including functions such as play, pause, fast forward, rewind, volume adjustment, and progress dragging. Applications can respond to user touch events input to the video container by calling these interfaces, enabling real-time switching of the video playback state. For example, when receiving a swipe gesture event from the touch interaction module, the video playback component parses the vector data of the gesture trajectory, calculates the swipe speed and direction, and maps it to the displacement of the progress bar, thereby precisely adjusting the video playback position.

[0065] In scenarios with multiple windows or scrollable interfaces, users can drag and drop the video container using touch controls to change its display position. For example... Figure 4 As shown, when the display position of the video container is about to exceed the screen area of ​​the display device 200 (over-screen state), the display device 200 discards the fixed aspect ratio constraint and, based on the extent to which it will exceed the screen area, irregularly reduces the size of the video container. Simultaneously, the video image displayed within the video container is also compressed (e.g., ...). Figure 4 (In the middle ab stage), the complete video frame is forcibly adapted into the adjusted container area.

[0066] However, the aforementioned video container processing method, due to the irregular compression of the video image, not only causes problems such as video aspect ratio distortion and detail loss, but also disrupts the correspondence between touch coordinates and the video image, resulting in touch operations failing to match the video image and causing touch failure (e.g., Figure 4 The issue (during the CD phase) affects the user experience.

[0067] To address the aforementioned issues, some embodiments of this application provide a display device 200 that dynamically calculates the visible area of ​​the video container when the video container extends beyond the screen area. Based on this visible area, the display device 200 determines the target rendering area in the original video's video coordinate system and renders the video image of the target rendering area using the video container. Simultaneously, the display device 200 establishes a bidirectional mapping between the screen coordinate system and the touch coordinate system, ensuring that touch operations are accurately mapped to the video content in the out-of-screen state, thus avoiding touch failure issues. Through this dynamic calculation of the target rendering area and bidirectional coordinate mapping mechanism, the display device 200 can guarantee the aspect ratio and display quality of the video content in the out-of-screen state, while also ensuring accurate touch response, thus optimizing the user experience of the display device 200.

[0068] In some embodiments, the display device 200 includes a display 260 and a controller 250. The display 260 is configured to display screen content rendered by a video container in a screen area according to a screen coordinate system. The video container is used to render the video frame of the original video. The display 260 is connected to a touch interaction module, which is used to receive touch events input by the user. The touch events include touch coordinates in a touch coordinate system. The controller 250 is coupled to the display 260 and the touch interaction module, and is configured to constrain the width and height dimensions of the video container according to a preset aspect ratio.

[0069] The video container is the display window used to hold and render video content. The width and height of the video container determine the display ratio of the video on the screen. The preset aspect ratio is a fixed ratio value set according to the original ratio of the video, such as 16:9, 4:3, 21:9, etc., to ensure that the video content will not be stretched or compressed during display.

[0070] Figure 5 This is a flowchart illustrating the process of constraining the width and height dimensions of a video container according to some embodiments of this application. Figure 5 As shown, in some embodiments, controller 250 is configured to perform the following steps: S501. Obtain the initial width and initial height of the video container.

[0071] In order to constrain the width and height of the video container by a preset aspect ratio, the display device 200 first obtains the default initial width (W0) and initial height (H0), and uses the initial width and initial height as a reference to determine the final container width and container height of the video container.

[0072] S502. Based on the initial width, calculate the corresponding container adaptation height according to the preset aspect ratio.

[0073] Based on the initial width, the corresponding height is calculated according to the preset aspect ratio and used as the container's adaptation height. For example, if the preset aspect ratio is r and the initial width is W0, the container's adaptation height H' = W0 / r.

[0074] S503. If the container adaptation height is less than or equal to the initial height, set the container width of the video container to the initial width and the container height of the video container to the container adaptation height.

[0075] After calculating the container adaptation height, if the container adaptation height is less than or equal to the initial height, it means there is enough space in the width direction. The initial width and the calculated container adaptation height can then be set as the width and height dimensions of the video container.

[0076] S504. If the container adaptation height is greater than the initial height, calculate the corresponding container adaptation width based on the preset aspect ratio, using the initial height as the reference; then set the container width of the video container to the container adaptation width, and set the container height of the video container to the initial height.

[0077] If the container's adapted height is greater than the initial height, it means there is not enough space in the height direction. In this case, the height is fixed, and the width is calculated based on the initial height and the preset aspect ratio to serve as the container's adapted width. Then, the container's adapted width and the initial height are set as the width and height dimensions of the video container.

[0078] Through the above steps S501-S504, the display device 200 can ensure that the video container always maintains a preset aspect ratio, preventing the video image rendered in the video container from being compressed or stretched, thus ensuring the display effect of the video image.

[0079] Figure 6 This is a flowchart illustrating a video playback processing method provided in some embodiments of this application. Figure 6 As shown, in some embodiments, controller 250 is configured to perform the following steps: S601. In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system.

[0080] Users can drag the video container on the screen using a long-press gesture to adjust its display position in real time. When the video container's display area exceeds the screen area, the overlapping area is mapped to the video container's container coordinate system to determine its visible area.

[0081] In some embodiments, when the display device 200 maps the overlapping area to the container coordinate system, it detects the left boundary (containerLeft) and top boundary (containerTop) of the video container in the screen coordinate system. A container coordinate system is established with the top-left corner (containerLeft, containerTop) of the video container as the origin, the direction to the right along the left boundary of the container as the positive direction of the horizontal coordinate, and the direction downward along the top boundary of the container as the positive direction of the vertical coordinate. Figure 7 (As shown).

[0082] In the container coordinate system, the maximum value between 0 and the negative left boundary of the container is determined as the visible left boundary (visibleLeft), the maximum value between 0 and the negative top boundary of the container is determined as the visible top boundary (visibleTop), the minimum value between the container width (containerWidth) and the first difference is determined as the visible right boundary (visibleRight), and the minimum value between the container height (containerHeight) and the second difference is determined as the visible bottom boundary (visibleBottom). Here, the first difference is the difference between the screen width (screenWidth) of the screen area and the left boundary of the container, and the second difference is the difference between the screen height (screenHeight) of the screen area and the top boundary of the container.

[0083] That is, the first difference is screenWidth-containerLeft, and the second difference is screenHeight-containerTop.

[0084] For example, such as Figure 8 As shown, the overlapping area in the screen coordinate system is mapped to the container coordinate system: visibleLeft=max(0,-containerLeft); visibleTop=max(0,-containerTop); visibleRight=min(containerWidth,screenWidth-containerLeft); visibleBottom=min(containerHeight,screenHeight-containerTop). The visible area of ​​the rectangle can be determined using the above visibleLeft, visibleTop, visibleRight, and visibleBottom.

[0085] In some embodiments, when obtaining the screen width and screen height of the screen area, the controller 250 of the display device 200 calls the system interface of the display 260 to directly read the physical resolution parameters of the screen; or it obtains the logical pixel size of the screen through the API provided by the operating system, wherein the logical pixel size needs to be converted in combination with the pixel density of the device. For multi-screen display scenarios, the target display screen also needs to be specified through the screen index to ensure that the accurate width and height data of the screen where the current video container is located are obtained.

[0086] S602. Based on the constraints of the preset aspect ratio, the video aspect ratio of the original video is maintained, and the visible area in the container coordinate system is transformed to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area.

[0087] After obtaining the visible area in the container coordinate system, the display device 200 keeps the aspect ratio of the original video unchanged based on the preset aspect ratio constraint for the video container. Under this premise, the visible area of ​​the container coordinate system is transformed to the video coordinate system of the original video, thereby obtaining the corresponding target rendering area. The target rendering area is the data basis for the video container to render the original video in the future.

[0088] At this point, because the video container is constrained by a preset aspect ratio, the preset processing strategy needs to process the original video while maintaining its original aspect ratio. That is, the original video's aspect ratio will also remain unchanged. Therefore, the video image rendered by the video container will not experience stretching or compression.

[0089] In order to adapt to the preset aspect ratio of the video container, in some embodiments, the display device 200 can calculate the target rendering area using two preset processing strategies, including cropping the original video based on the visible area (preset processing strategy for cropping), and scaling the original video based on the visible area (preset processing strategy for scaling).

[0090] Regarding the cropping method, in some embodiments, when calculating the target rendering area, the display device 200 reads the original video height (videoHeight) and original video width (videoWidth) of the original video, and reads the container height (containerHeight) and container width (containerWidth) of the video container. Then, it calculates the height ratio of the original video height to the container height as the height scaling factor (scaleY); and calculates the width ratio of the original video width to the container width as the width scaling factor (scaleX).

[0091] In the video coordinate system, the product of the visible left boundary and the width ratio of the visible area is determined as the visible video left boundary (videoLeft'), the product of the visible top boundary and the height ratio of the visible area is determined as the visible video top boundary (videoTop'), the product of the width and the width ratio of the visible area is determined as the visible video width (videoWidth'), and the product of the height and the height ratio of the visible area is determined as the visible video height (videoHeight').

[0092] For example, scaleX=videoWidth / containerWidth; scaleY=videoHeight / containerHeight; videoLeft'=visibleLeft×scaleX; videoTop'=visibleTop×scaleY; videoWidth'=visibleWidth×scaleX; videoHeight'=visibleHeight×scaleY.

[0093] Next, a corresponding cropping window is generated based on the visible video's left and top boundaries, width, and height. This cropping window is then used to crop the original video, resulting in the target rendering area in the video coordinate system. In other words, based on the calculated visible video's left and top boundaries, width, and height, a corresponding rectangular region is determined within the original video. This determined rectangular region is then cropped to obtain the target rendering area for subsequent rendering. This method avoids pixel compression and stretching issues in the target rendering area obtained through cropping.

[0094] In some embodiments, when the display device 200 generates the cropping window, it also determines the maximum value between 0 and a first reference value as the cropping left boundary (cropLeft), the maximum value between 0 and a second reference value as the cropping top boundary (cropTop), the minimum value between 0 and a third reference value as the cropping width (cropWidth), and the maximum value between 0 and a fourth reference value as the cropping height (cropHeight) in the video coordinate system. Then, a rectangular object is created according to the cropping left boundary, cropping top boundary, cropping width, and cropping height to serve as the cropping window.

[0095] The first reference value is the minimum value between the original video width and the left edge of the visible video; the second reference value is the minimum value between the original video height and the top edge of the visible video; the third reference value is the minimum value between the third difference and the width of the visible video, and the third difference is the difference between the original video width and the left edge of the crop; the fourth reference value is the minimum value between the fourth difference and the height of the visible video, and the fourth difference is the difference between the original video height and the top edge of the crop.

[0096] For example, cropLeft=max(0,min(videoWidth,videoLeft')); cropTop=max(0,min(videoHeight,videoTop')); cropWidth=max(0,min(videoWidth–cropLeft,videoWidth')); cropHeight=max(0,min(videoHeight–cropTop,videoHeight')).

[0097] Based on the above embodiments, the target rendering area calculated by the display device 200 in the video coordinate system by the cropping method has a left boundary of cropLeft, a top boundary of cropTop, a width of cropWidth, and a height of cropHeight.

[0098] Regarding the scaling method, in some embodiments, when the display device 200 calculates the target rendering area, it reads the original video height and original video width of the original video, as well as the visible area width and visible area height of the visible area.

[0099] In some implementations, the visible area width (visibleWidth) is the maximum value between 0 and the visible right boundary minus the visible left boundary, i.e., max(0,visibleRight-visibleLeft), and the visible area height (visibleHeight) is the maximum value between 0 and the visible bottom boundary minus the visible top boundary, i.e., max(0,visibleBottom-visibleTop).

[0100] Then, the ratio of the original video width to the original video height is calculated to obtain the video aspect ratio (r'). Based on the height of the visible area, the corresponding video adaptation width (targetWidth) is calculated according to the video aspect ratio, i.e., targetWidth = visibleHeight × r'.

[0101] The display device 200 calculates the final size of the original video within the visible area, using the shortest side of the visible area as a constraint to maintain the initial aspect ratio of the video. In other words, after calculating the video adaptation width, the display device 200 compares the video adaptation width with the visible area width. If the video adaptation width is less than or equal to the visible area width (targetWidth ≤ visibleWidth), the target rendering area of ​​the original video is determined based on the visible area height and the video adaptation width, following the principle of centering the display. That is, the visible area height is used as the final height (finalHeight) of the target rendering area, and the video adaptation width is used as the final width (finalWidth) of the target rendering area.

[0102] In some implementations, when the display device 200 determines the target rendering area of ​​the original video according to the principle of centering display based on the visible area height and the video adaptation width, in the video coordinate system, the left boundary of the target rendering area is set to half the difference between the visible area width and the video adaptation width, i.e., (visibleWidth – finalWidth) / 2; and the upper boundary of the target rendering area is set to 0.

[0103] If the video's adaptive width is greater than the visible area width (targetWidth > visibleWidth), the corresponding video adaptive height (targetHeight) is calculated based on the visible area width and the video's aspect ratio. The original video is then scaled according to the ratio of the visible area width to the video adaptive height, and the target rendering area of ​​the scaled original video is determined according to the principle of centering. In other words, the height of the scaled video is used as the final height (finalHeight) of the target rendering area, and the width of the scaled video is used as the final width (finalWidth) of the target rendering area.

[0104] Similarly, in some implementations, when the display device 200 determines the target rendering area of ​​the scaled original video according to the principle of centered display, it reads the target video width (finalWidth) of the scaled original video in the video coordinate system. It calculates half the difference between the visible area width and the target video width, i.e., (visibleWidth – finalWidth) / 2. Then, in the video coordinate system, it sets this half value as the left boundary of the target rendering area and sets the upper boundary of the target rendering area to 0.

[0105] S603, Render the video image of the target rendering area through the video container.

[0106] After calculating the corresponding target rendering area based on the visible area using a preset method (cropping or scaling), the display device 200 can render the video image of the target rendering area through a video container, so that the monitor 260 can display the corresponding video image in the target rendering area through the video container.

[0107] Since the above cropping method involves cropping the original video, the display device 200 does not need to change the layout parameters when rendering the video frame in the target rendering area (see the area of ​​the original video). It can maintain the original layout parameters, i.e., layoutLeft=cropLeft; layoutTop=cropTop; layoutRight=cropLeft+cropWidth; layoutBottom=cropTop+cropHeight.

[0108] In other words, after the display device 200 crops the original video, the left boundary layout parameter (layoutLeft) of the video container is the left boundary of the target rendering area after cropping, the top boundary layout parameter (layoutTop) of the video container is the top boundary of the target rendering area after cropping, the right boundary layout parameter (layoutRight) of the video container is the right boundary of the target rendering area after cropping, and the bottom boundary layout parameter (layoutBottom) of the video container is the bottom boundary of the target rendering area after cropping. For example, if the video container exceeds the screen area, the display device 200 will... Figure 9 The original video is cropped in the manner shown so that the corresponding video image is presented through the target rendering area 901.

[0109] Regarding the scaling method, in some embodiments, when the display device 200 renders the video image of the target rendering area through the video container, it updates the left boundary layout parameter (layoutLeft) of the video container to the sum of the visible left boundary of the visible area and the left boundary (finalLeft) of the target rendering area, and updates the top boundary layout parameter (layoutTop) of the video container to the sum of the visible top boundary of the visible area and the top boundary (finalTop) of the target rendering area. It can be understood that the left and top boundaries of the target rendering area referred to here are videoLeft and videoTop, obtained through the above embodiments after size calculation or scaling of the original video, and are used here for distinction.

[0110] Update the right boundary layout parameter (layoutRight) of the video container to the sum of the left boundary layout parameter and the target video width (finalWidth), and update the bottom boundary layout parameter (layoutBottom) of the video container to the sum of the top boundary layout parameter and the target video height (finalHeight) of the scaled original video.

[0111] For example, layoutLeft=visibleLeft+finalLeft; layoutTop=visibleTop+finalTop; layoutRight=layoutLeft+finalWidth; layoutBottom=layoutTop+finalHeight.

[0112] After updating the layout parameters of the video container, the display device 200 renders the video image of the target rendering area according to the left boundary layout parameters, top boundary layout parameters, right boundary layout parameters, and bottom boundary layout parameters of the video container. For example, if the video container exceeds the screen area, the display device 200 renders the video image according to the left boundary layout parameters, top boundary layout parameters, right boundary layout parameters, and bottom boundary layout parameters of the video container. Figure 10 The original video is scaled and centered at the top as shown, so that the corresponding video image is presented through the target rendering area 1001.

[0113] S604. Based on the visible area and the target rendering area, construct a two-way mapping relationship between the screen coordinate system and the touch coordinate system, so as to respond to touch events for the target rendering area according to the two-way mapping relationship.

[0114] The display device 200 renders the video image corresponding to the target rendering area through a video container. Simultaneously, it performs a reversible mapping from the screen coordinate system to the touch coordinate system based on the visible area and the target rendering area to achieve precise matching of touch events and improve touch failure issues. For example, the display device 200 can construct a bidirectional mapping relationship through a two-dimensional affine transformation.

[0115] In some embodiments, the process of establishing a bidirectional mapping relationship involves coordinate system transformation and calibration. In the screen coordinate system, the boundary of the visible area is determined by visibleLeft, visibleTop, visibleRight, and visibleBottom, while the boundary of the target rendering area in the container coordinate system is determined by layoutLeft, layoutTop, layoutRight, and layoutBottom. The display device 200 first calculates the offset and scaling ratio between the screen coordinate system and the container coordinate system, establishing a positive mapping relationship from screen coordinates (x_screen, y_screen) to container coordinates (x_container, y_container).

[0116] The formula for forward mapping is: x_container = (x_screen - visibleLeft) × (layoutRight - layoutLeft) / (visibleRight - visibleLeft) + layoutLeft; y_container = (y_screen - visibleTop) × (layoutBottom - layoutTop) / (visibleBottom - visibleTop) + layoutTop. Forward mapping converts the actual landing point of the user's touch operation on the screen into the corresponding position in the container coordinate system, ensuring that the touch event can accurately locate the specific element within the target rendering area.

[0117] Accordingly, the inverse mapping relationship is used to transform the touch response area in the container coordinate system back to the screen coordinate system for visual feedback or boundary detection on the screen. The inverse mapping calculation formula is: x_screen=(x_container-layoutLeft)×(visibleRight-visibleLeft) / (layoutRight-layoutLeft)+visibleLeft; y_screen=(y_container-layoutTop)×(visibleBottom-visibleTop) / (layoutBottom-layoutTop)+visibleTop.

[0118] When the display device 200 receives a touch event, it first converts the screen touch coordinates to container coordinates through forward mapping and determines whether the coordinates of the touch event fall within the target rendering area. If it is determined to be a valid touch, it executes the corresponding business logic processing based on the container coordinates, such as video playback control and interface element interaction. After processing, if it is necessary to display the touch feedback effect on the screen, it converts the container coordinates back to screen coordinates through reverse mapping for rendering output.

[0119] In this way, the display device 200 can accurately match touch events with corresponding response programs through a two-way mapping relationship, avoiding touch failure problems and improving the accuracy of touch operation.

[0120] To facilitate rapid processing of video playback in the above-mentioned over-screen state by the display device 200, in some embodiments, the display device 200 also records the associated parameters of the visible area, the target rendering area, and the bidirectional mapping relationship, such as the parameters of the visible area, the parameters of the target rendering area, the layout parameters of the video container, and the parameters of the bidirectional mapping relationship. When the display device 200 detects that the video container exceeds the screen area again, in response to the video container's display area exceeding the screen area, if the difference between the current visible area and the recorded visible area is less than a preset threshold (e.g., 2-5 pixels), the associated parameters are reused, the video image of the target rendering area is rendered according to the associated parameters, and the touch events for the target rendering area are responded to according to the associated parameters to achieve a fast response effect, thereby optimizing the device performance of the display device 200.

[0121] Figure 11 Interactive diagrams illustrating video playback processing methods provided in some embodiments of this application. In some embodiments, such as... Figure 11As shown, display device 200 renders the original video frame through a video container, and controller 250 constrains the aspect ratio of the video container to remain constant through a preset aspect ratio. Display device 200 receives touch events input by the user to move the video container's position via a touch interaction module. Controller 250 responds to these touch events by changing the display position of the video container, causing the position of the video container displayed on monitor 260 to update synchronously. When the display area of ​​the video container exceeds the screen area, controller 250 maps the overlapping area of ​​the video container and the screen area to the container coordinate system (the process of determining the visible area), obtaining the visible area. Then, controller 250 maintains the preset aspect ratio of the video container and, based on the visible area, processes the original video using either cropping (cropping the original video) or scaling (scaling the original video and centering it) to calculate the corresponding target rendering area. The video frame of the target rendering area is then rendered through the video container, and a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed based on the visible area and the target rendering area. This bidirectional mapping relationship is used to respond to touch events targeting the target rendering area. When the touch interaction module receives a touch event from the user targeting the rendering area, it responds to the touch event according to the bidirectional mapping relationship.

[0122] It should be noted that, Figure 11 The cropping or scaling (centering display) processing method described above can be referred to the above embodiments, and will not be repeated here.

[0123] For example, the target rendering area calculated under the cropping method, and the display effect of the display device 200 when the user moves the position of the video container are as follows: Figure 12 As shown. The target rendering area is calculated under scaling mode. When the user moves the video container, the display effect on display device 200 is as follows. Figure 13 As shown.

[0124] In some embodiments, when determining whether the display area of ​​the video container exceeds the screen area, the display device 200 may base its determination on the left boundary, top boundary, width, and height of the video container in the screen coordinate system. If the left boundary of the video container in the screen coordinate system is greater than or equal to 0, and the top boundary is greater than or equal to 0, and the sum of the left boundary and the width is less than or equal to the screen width, and the sum of the top boundary and the height is less than or equal to the screen height, then the display area of ​​the video container is determined to be completely within the screen area; otherwise, it exceeds the screen area.

[0125] For example, if containerLeft>=0&&containerTop>=0&&containerLeft+containerWidth<=screenWidth&&containerTop+containerHeight<=screenHeight, then the video container is completely within the screen area; otherwise, it is determined that the display area of ​​the video container exceeds the screen area.

[0126] In some embodiments, when the video container is completely within the screen area, the display device 200, when rendering the video frame of the original video through the video container, performs a mapping from screen coordinates to video coordinates based on the screen width, screen height, and video width and video height, so as to achieve playback and display based on the video container without exceeding the screen limit.

[0127] For example, calculate the scaling ratio from the screen to the original video: scaleX = videoWidth / screenWidth; scaleY = videoHeight / screenHeight. The left boundary of the video container is mapped to the left boundary of the video cropping: videoLeft = containerLeft × scaleX; videoTop = containerTop × scaleY. The right boundary of the video container is mapped to the right boundary of the video cropping, and the bottom boundary is mapped to the bottom boundary of the video cropping: videoRight = (containerLeft + containerWidth) × scaleX; videoBottom = (containerTop + containerHeight) × scaleY. Calculate the size of the area to be cropped from the original video: videoCropWidth = videoRight – videoLeft; videoCropHeight = videoBottom – videoTop. To prevent cropping coordinates from exceeding the video boundaries: cropLeft=max(0,min(videoWidth,videoLeft)); cropTop=max(0,min(videoHeight,videoTop)); cropWidth=max(0,min(videoWidth–cropLeft,videoCropWidth)); cropHeight=max(0,min(videoHeight–cropTop,videoCropHeight)).

[0128] Based on the above coordinate mapping and cropping boundary calculation, the display device 200 extracts valid image data from the video stream of the original video according to the cropping area determined by cropLeft, cropTop, cropWidth, and cropHeight. Then, the graphics processor scales the cropped video frames to the actual display size of the video container and finally outputs them to the corresponding position on the screen to complete the presentation.

[0129] It should be noted that the left, right, top, and bottom boundaries of the aforementioned video container, original video, and cropping window can be determined based on at least one of the corresponding top-left and bottom-right corner coordinates. For example, the left boundary is the horizontal coordinate of the top-left corner and the top boundary is the vertical coordinate of the top-left corner. The right and bottom boundaries are then determined by combining the width and height. This application does not impose any restrictions on this.

[0130] In this application embodiment, the original video can be video stream data provided by any signal source, including but not limited to picture-in-picture, screen casting, streaming media, video captured by a camera, television signal, or video data obtained from an external device through other means, and this application does not impose any restrictions on it.

[0131] Based on the above embodiments, this application also provides a video playback processing method, which can be applied to the above-mentioned display device 200. The display 260 of the display device 200 is configured to display the image content rendered by the video container in the screen area according to the screen coordinate system. The video container is used to render the video image of the original video. The display 260 is connected to a touch interaction module, which is used to receive touch events input by the user. The touch events include touch coordinates in the touch coordinate system. The display device 200 constrains the width and height dimensions of the video container according to a preset aspect ratio. Figure 6 As shown, the method includes the following steps: S601. In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system.

[0132] S602. Based on the preset aspect ratio constraint, maintain the video aspect ratio of the original video, and transform the visible area in the container coordinate system to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area.

[0133] S603, Render the video image of the target rendering area through the video container.

[0134] S604. Based on the visible area and the target rendering area, construct a two-way mapping relationship between the screen coordinate system and the touch coordinate system, so as to respond to touch events for the target rendering area according to the two-way mapping relationship.

[0135] Based on the above video playback processing method, the size of the video container is always constrained by a preset aspect ratio to avoid video stretching and compression caused by irregular container adjustments. By accurately mapping the visible area and calculating the target rendering area, it is ensured that only the effective video image is rendered when the video container exceeds the screen, preventing image aspect ratio imbalance and detail distortion. At the same time, a two-way mapping relationship between the screen and the touch coordinate system is constructed so that the touch coordinates can accurately match the interactive area of ​​the video image, solving the touch failure problem when the video exceeds the screen and optimizing the visual effect of video playback and the touch interaction experience.

[0136] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that, include: The display is configured to display screen content rendered by a video container in a screen area according to a screen coordinate system, the video container being used to render the video frame of the original video. The display is connected to a touch interaction module, which is used to receive touch events input by the user. The touch events include touch coordinates in the touch coordinate system. A controller is coupled to the display and the touch interaction module; the controller is configured to constrain the width and height dimensions of the video container according to a preset aspect ratio, and is further configured to: In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system; Based on the preset aspect ratio constraint, the video aspect ratio of the original video is maintained, and the visible area in the container coordinate system is transformed to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area. The video image of the target rendering area is rendered through the video container; Based on the visible area and the target rendering area, a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed to respond to touch events targeting the target rendering area according to the bidirectional mapping relationship.

2. The display device according to claim 1, characterized in that, The controller executes the constraint of the width and height dimensions of the video container according to a preset aspect ratio, specifically configured as follows: Obtain the initial width and initial height of the video container; Based on the initial width, the corresponding container adaptation height is calculated according to the preset aspect ratio; If the container adaptation height is less than or equal to the initial height, the container width of the video container is set to the initial width, and the container height of the video container is set to the container adaptation height. If the container adaptation height is greater than the initial height, the corresponding container adaptation width is calculated based on the initial height and the preset aspect ratio; then the container width of the video container is set to the container adaptation width, and the container height of the video container is set to the initial height.

3. The display device according to claim 2, characterized in that, The controller is configured to map the overlapping area between the video container and the screen area to the container coordinate system of the video container. Detect the left and top boundaries of the video container in the screen coordinate system; A container coordinate system is established with the top left corner of the video container as the origin, the direction to the right along the left boundary of the container as the positive direction of the horizontal coordinate, and the direction downward along the top boundary of the container as the positive direction of the vertical coordinate. In the container coordinate system, the maximum value between 0 and the negative left boundary of the container is determined as the visible left boundary of the visible area, the maximum value between 0 and the negative top boundary of the container is determined as the visible top boundary of the visible area, the minimum value between the container width and a first difference is determined as the visible right boundary of the visible area, and the minimum value between the container height and a second difference is determined as the visible bottom boundary of the visible area; wherein, the first difference is the difference between the screen width of the screen area and the left boundary of the container, and the second difference is the difference between the screen height of the screen area and the top boundary of the container.

4. The display device according to any one of claims 1-3, characterized in that, The controller performs a preset processing strategy to transform the visible area in the container coordinate system to the video coordinate system of the original video, specifically configured as follows: Read the original video height and original video width of the original video, and read the container height and container width of the video container; Calculate the height ratio of the original video height to the container height, and calculate the width ratio of the original video width to the container width; In the video coordinate system, the product of the visible left boundary corresponding to the visible area and the width ratio is determined as the visible video left boundary, the product of the visible upper boundary corresponding to the visible area and the height ratio is determined as the visible video upper boundary, the product of the visible width corresponding to the visible area and the width ratio is determined as the visible video width, and the product of the visible height corresponding to the visible area and the height ratio is determined as the visible video height. A cropping window is generated based on the left boundary of the visible video, the top boundary of the visible video, the width of the visible video, and the height of the visible video; The original video is cropped using the cropping window to obtain the target rendering area in the video coordinate system.

5. The display device according to claim 4, characterized in that, The controller generates a cropping window based on the left edge, top edge, width, and height of the visible video, specifically configured as follows: In the video coordinate system, the maximum value between 0 and the first reference value is determined as the left cropping boundary, the maximum value between 0 and the second reference value is determined as the top cropping boundary, the minimum value between 0 and the third reference value is determined as the cropping width, and the maximum value between 0 and the fourth reference value is determined as the cropping height; wherein, the first reference value is the minimum value between the original video width and the visible video left boundary, the second reference value is the minimum value between the original video height and the visible video top boundary, the third reference value is the minimum value between the third difference and the visible video width, the third difference is the difference between the original video width and the cropping left boundary, the fourth reference value is the minimum value between the fourth difference and the visible video height, and the fourth difference is the difference between the original video height and the cropping top boundary; A rectangular object is created based on the left cropping boundary, the top cropping boundary, the cropping width, and the cropping height to serve as the cropping window.

6. The display device according to any one of claims 1-3, characterized in that, The controller performs a preset processing strategy to transform the visible area in the container coordinate system to the video coordinate system of the original video, specifically configured as follows: Read the original video height and original video width of the original video, and read the visible area width and visible area height of the visible area; Calculate the ratio of the original video width to the original video height to obtain the video aspect ratio; Based on the height of the visible area, the corresponding video adaptation width is calculated according to the video aspect ratio; If the video adaptation width is less than or equal to the visible area width, the target rendering area of ​​the original video is determined according to the visible area height and the video adaptation width, based on the principle of centering display. If the video adaptation width is greater than the visible area width, the corresponding video adaptation height is calculated based on the visible area width and the video aspect ratio. The original video is then scaled according to the ratio of the visible area width to the video adaptation height, and the target rendering area of ​​the scaled original video is determined according to the principle of center display.

7. The display device according to claim 6, characterized in that, The controller determines the target rendering area of ​​the scaled original video according to the principle of centering display, and is specifically configured as follows: Under the video coordinate system, read the target video width of the original video after scaling; Calculate half the difference between the width of the visible area and the width of the target video; In the video coordinate system, the half value is set as the left boundary of the target rendering area, and the upper boundary of the target rendering area is set to 0.

8. The display device according to claim 7, characterized in that, The controller is configured to render the video frame of the target rendering area through the video container, specifically as follows: The left boundary layout parameter of the video container is updated to the sum of the visible left boundary of the visible area and the left boundary of the target rendering area; the upper boundary layout parameter of the video container is updated to the sum of the visible upper boundary of the visible area and the upper boundary of the target rendering area. The right boundary layout parameter of the video container is updated to the sum of the left boundary layout parameter and the target video width, and the lower boundary layout parameter of the video container is updated to the sum of the upper boundary layout parameter and the target video height of the original video after scaling. The video image of the target rendering area is rendered using the video container according to the left boundary layout parameters, the upper boundary layout parameters, the right boundary layout parameters, and the lower boundary layout parameters.

9. The display device according to claim 1, characterized in that, The controller is also configured to: Record the associated parameters of the visible area, the target rendering area, and the bidirectional mapping relationship; In response to the video container's display area exceeding the screen area, if the difference between the currently visible area and the recorded visible area is less than a preset threshold, the associated parameters are reused. Render the video image of the target rendering area according to the associated parameters, and respond to touch events for the target rendering area according to the associated parameters.

10. A method for processing video playback, characterized in that, Applied to display devices, the display of the display device is configured to display screen content rendered by a video container in a screen area according to a screen coordinate system, wherein the video container is used to render the video frame of the original video. The display is connected to a touch interaction module, which receives touch events input by the user. The touch events include touch coordinates in a touch coordinate system. The display device constrains the width and height dimensions of the video container according to a preset aspect ratio. The method includes: In response to the display area of ​​the video container exceeding the screen area, the overlapping area of ​​the video container and the screen area is mapped to the container coordinate system of the video container to obtain the visible area of ​​the video container in the container coordinate system; Based on the preset aspect ratio constraint, the video aspect ratio of the original video is maintained, and the visible area in the container coordinate system is transformed to the video coordinate system of the original video according to the preset processing strategy to obtain the target rendering area. The video image of the target rendering area is rendered through the video container; Based on the visible area and the target rendering area, a bidirectional mapping relationship between the screen coordinate system and the touch coordinate system is constructed to respond to touch events targeting the target rendering area according to the bidirectional mapping relationship.

Images