Display device and screen projection method

Abstract

The application provides a display device, comprising a display, a rotating assembly, and a controller configured to receive image information sent by a terminal, wherein when the terminal is in a portrait mode, the image information comprises valid information and left and right black information, the valid information corresponds to screen display content of the terminal, and the controller is configured to control the display to present a projection picture based on the valid information, wherein the projection picture is obtained by magnifying the valid information by a preset multiple.

Description

[0001] This application claims priority to Chinese Patent Application No. 202010334727.9, filed on April 24, 2020, entitled "A Display Device and Screen Projection Method", and to Chinese Patent Application No. 202010331501.3, filed on April 24, 2020, entitled "A Screen Projection Video Stream Effective Resolution Detection Method and Display Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of smart TV technology, and in particular to a display device and a screen projection method. Background Technology

[0003] Screen mirroring is an interactive operation between a terminal and a display device. It typically uses a wireless local area network (WLAN) to transmit a video stream, allowing the display device to show the screen on the terminal device. Taking mobile phone screen mirroring as an example, for a mobile phone and a smart TV connected to the same Wi-Fi network, the screen mirroring command can be executed on the mobile phone to send the screen displayed on the mobile phone as a video stream to the smart TV, utilizing the large screen of the smart TV for a better user experience.

[0004] However, the aspect ratio of mobile phones and other terminals often differs from that of display devices. For example, under normal operation, a mobile phone screen has an aspect ratio of 1080:1940, while a smart TV monitor has an aspect ratio of 1940:1080. This means the mobile phone screen is vertically oriented, while the smart TV screen is horizontally oriented. Therefore, when casting a mobile phone screen to a smart TV, the screen may not display correctly due to the mismatch between the mobile phone's aspect ratio and the smart TV's aspect ratio.

[0005] To fully display the screen on a mobile phone, the image needs to be scaled based on the phone's height. However, when scaling the projected image, the difference in aspect ratio results in large black areas on both sides of the smart TV screen, which not only reduces the user's viewing experience but also wastes display space. Summary of the Invention

[0006] This application provides a display device and a projection method to solve the problem of wasting display space on the screen when displaying projected images on traditional display devices.

[0007] On one hand, this application provides a display device, characterized in that it includes:

[0008] monitor;

[0009] A rotating component is configured to rotate the display so that the display is in a rotation state, either in landscape or portrait mode.

[0010] User interface, wherein the communication information is configured to connect to a terminal;

[0011] The controller is configured as follows:

[0012] The system receives image information sent by the terminal, wherein when the terminal is in portrait mode, the image information includes valid information and left and right black areas;

[0013] If the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to portrait mode;

[0014] Based on the effective information, the display is controlled to present the projected image, wherein the projected image is obtained by magnifying the effective information by a preset factor.

[0015] On the other hand, this application also provides a screen projection method applied to a display device, characterized by comprising:

[0016] The terminal receives image information sent by the terminal, wherein when the terminal is in portrait mode, the image information includes valid information and left and right black information, and the valid information corresponds to the screen display content of the terminal;

[0017] If the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to portrait mode;

[0018] Based on the effective information, the display is controlled to present the projected image, wherein the projected image is obtained by magnifying the effective information by a preset factor.

[0019] As can be seen from the above technical solutions, the display device and projection method provided in the first aspect of this application allow the display controller to receive image information sent by the terminal. When the terminal is in portrait mode, the image information includes valid information and left and right black information. If the current rotation state of the display does not match the portrait mode of the terminal, the display is rotated to portrait mode. Based on the valid information, the display is controlled to present the projection image, wherein the projection image is obtained by magnifying the valid information by a preset factor. The display device provided in this application can determine whether the mobile terminal is in portrait mode based on image information and automatically adjust the rotation state of the display, thereby using a larger display space to display the projection image and alleviating the problem that traditional smart TVs cannot display projection images normally. Attached Figure Description

[0020] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A This is an application scenario diagram of a display device according to this application;

[0022] Figure 1B This is a rear view of a display device according to this application;

[0023] Figure 2 This is a hardware configuration block diagram of the control device of this application;

[0024] Figure 3 This application shows a block diagram illustrating the architecture configuration of the operating system in the device memory.

[0025] Figure 4A This is a schematic diagram of the landscape mode of the mobile terminal in this application;

[0026] Figure 4B This is a schematic diagram of the vertical mode of the mobile terminal in this application;

[0027] Figure 5A This is a schematic diagram illustrating how a mobile terminal in landscape mode sends image information to a display device in landscape mode, according to one embodiment of this application.

[0028] Figure 5B This is a schematic diagram illustrating how a mobile terminal in landscape mode sends image information to a display device in portrait mode, according to one embodiment of this application.

[0029] Figure 6A This is a schematic diagram illustrating how a mobile terminal in portrait mode sends image information to a display device in landscape mode, according to one embodiment of this application.

[0030] Figure 6B This is a schematic diagram illustrating the effect of displaying the image in portrait mode on a screen mirroring device according to the screen mirroring protocol.

[0031] Figure 6C This is a schematic diagram illustrating how a mobile terminal in portrait mode sends image information to a display device in portrait mode in one embodiment of this application.

[0032] Figure 7 This is a schematic diagram of the effective resolution detection method for screen-projected video stream according to this application;

[0033] Figure 8 This is a schematic diagram illustrating the process of comparing the sampling resolution with the initial resolution in this application;

[0034] Figure 9AThis is a flowchart illustrating the calculation of the baseline and comparison values ​​in this application;

[0035] Figure 9B This is a schematic diagram of the black border data and initial screen data of this application;

[0036] Figure 10 This is a flowchart illustrating the process of determining the preset execution conditions in this application;

[0037] Figure 11 This is a flowchart illustrating the process of comparing multiple sampling resolutions and an initial resolution in this application;

[0038] Figure 12 This is a schematic diagram illustrating the process executed by the controller of an exemplary display device according to this application;

[0039] Figure 13 This is a flowchart illustrating the process of determining the effective information magnification factor based on the ratio of the height and width of the display device and the effective screen in this application.

[0040] Figure 14 This is a schematic diagram of the structure of a display device according to this application;

[0041] Figures 15A-15C This is a schematic diagram of the user interface of a display device receiving a projected image in portrait mode according to the present application. Detailed Implementation

[0042] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0043] In the technical solution provided in this application, the display device can be an electrical appliance with a large screen, such as a smart TV, that presents video and audio signals to the user. The display device can have an independent operating system and support functional expansion. Various applications can be installed on the display device according to user needs, such as traditional video applications, social applications like short videos, and reading applications like comics and books. These applications can utilize the display device's screen to display application content, providing users with richer media resources. Simultaneously, the display device can also interact with different terminals and share resources. For example, a smart TV can connect to a mobile phone via wireless communication methods such as a local area network or Bluetooth, thereby playing resources from the mobile phone or directly projecting the screen from the mobile phone.

[0044] To facilitate the display of target media asset details pages in different screen orientations (portrait and landscape) and improve the user viewing experience, this application provides a display device and a computer storage medium. The display device, such as a rotating television, is applicable not only to rotating televisions but also to other display devices, such as computers and tablets.

[0045] The term "module" as used in the embodiments of this application can refer to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0046] The term "remote control" as used in the embodiments of this application refers to a component of an electronic device (such as the display device disclosed in this application) that can wirelessly control the electronic device over a relatively short distance. This component typically connects to the electronic device using infrared and / or radio frequency (RF) signals and / or Bluetooth, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a handheld touch remote control replaces most of the physical built-in hard buttons in a typical remote control device with a user interface on a touchscreen.

[0047] The term "gesture" as used in the embodiments of this application refers to user behavior in which a user expresses an expected idea, action, purpose, and / or result through a change in hand shape or hand movement.

[0048] The term "hardware system" as used in the embodiments of this application can refer to a physical component with computing, control, storage, input, and output functions, composed of mechanical, optical, electrical, and magnetic devices such as integrated circuits (ICs) and printed circuit boards (PCBs). In the embodiments of this application, the hardware system is also commonly referred to as a motherboard, main chip, or controller.

[0049] See Figure 1A Figure 1 illustrates an application scenario of a display device provided in some embodiments of this application. As shown in Figure 1, the control device 100 and the display device 200 can communicate via wired or wireless means.

[0050] The control device 100 is configured to control the display device 200. It can receive operation commands input by the user and convert them into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200. For example, if the user operates the channel up / down keys on the control device 100, the display device 200 will respond with the channel up / down operation.

[0051] The control device 100 can be a remote control 100A, including infrared or Bluetooth communication protocols, and other short-range communication methods, to control the display device 200 wirelessly or via wired means. Users can input commands through buttons on the remote control, voice input, or control panel input to control the display device 200. For example, users can input corresponding control commands through volume up / down buttons, channel control buttons, up / down / left / right movement buttons, voice input buttons, menu buttons, and power buttons on the remote control to control the display device 200.

[0052] The control device 100 can also be a smart device, such as a mobile terminal 100B, tablet computer, computer, laptop computer, etc. For example, an application running on the smart device can be used to control the display device 200. This application can be configured to provide various controls to the user through an intuitive user interface (UI) on the screen associated with the smart device.

[0053] For example, mobile terminal 100B can install software applications with display device 200 to connect and communicate via network communication protocols, achieving one-to-one control operations and data communication. For instance, mobile terminal 100B can establish a control command protocol with display device 200, allowing operation of various function keys or virtual controls on the user interface provided on mobile terminal 100B to achieve functions similar to the physical buttons on remote control 100A. Audio and video content displayed on mobile terminal 100B can also be transmitted to display device 200 for synchronized display.

[0054] Display device 200 provides network television functionality, including broadcast reception and computer support. The display device can be implemented as digital television, network television, Internet Protocol television (IPTV), etc.

[0055] Display device 200 can be a liquid crystal display, an organic light-emitting display, or a projection device. Specific display device type, size, and resolution are not limited.

[0056] Display device 200 also communicates with server 300 via various communication methods. This allows display device 200 to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. Server 300 can provide display device 200 with various content and interactive features. For example, display device 200 can send and receive information, such as receiving Electronic Program Guide (EPG) data, receiving software updates, or accessing a remotely stored digital media library. Server 300 can be a group or multiple groups, and can be one or more types of servers. Other network services such as video-on-demand and advertising services can be provided through server 300.

[0057] In some embodiments, such as Figure 1B As shown, the display device 200 includes a rotating assembly 276, a controller 250, a display 275, a terminal interface 278 extending from a gap in the back panel, and the rotating assembly 276 connected to the back panel. The rotating assembly 276 allows the display 275 to rotate. From the perspective of viewing the display device from the front, the rotating assembly 276 can rotate the display to a portrait mode, i.e., a mode where the vertical side of the screen is longer than the horizontal side, or it can rotate the screen to a landscape mode, i.e., a mode where the horizontal side of the screen is longer than the vertical side.

[0058] Figure 2 The diagram illustrates, for example, a hardware configuration block diagram of a display device 200. (As shown...) Figure 2 As shown, the display device 200 may include a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a memory 260, a user interface 265, a video processor 270, a display 275, a rotating component 276, an audio processor 280, an audio output interface 285, and a power supply 290.

[0059] The rotating assembly 276 may include components such as a drive motor and a rotating shaft. The drive motor can be connected to the controller 250 and is controlled by the controller 250 to output a rotation angle. One end of the rotating shaft is connected to the power output shaft of the drive motor, and the other end is connected to the display 275, so that the display 275 can be fixedly mounted on a wall or bracket through the rotating assembly 276.

[0060] The rotating assembly 276 may also include other components, such as transmission components and detection components. The transmission component can adjust the rotational speed and torque output by the rotating assembly 276 through a specific transmission ratio, and can be a gear transmission. The detection component can consist of sensors mounted on the rotating shaft, such as angle sensors and attitude sensors. These sensors can detect parameters such as the rotation angle of the rotating assembly 276 and send the detected parameters to the controller 250, so that the controller 250 can determine or adjust the state of the display device 200 based on the detected parameters. In practical applications, the rotating assembly 276 may include, but is not limited to, one or more of the above-mentioned components.

[0061] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means and can perform modulation and demodulation processes such as amplification, mixing, and resonance. It is used to demodulate the audio and video signals carried in the frequency of the television channel selected by the user from multiple wireless or wired broadcast television signals, as well as additional information (such as EPG data).

[0062] The tuner 210, as selected by the user and controlled by the controller 250, responds to the frequency of the television channel selected by the user and the television signal carried by that frequency.

[0063] The tuner / demodulator 210 can receive signals through various means depending on the broadcasting system of the television signal, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, or internet broadcasting; and can use digital modulation or analog modulation depending on the modulation type; and can demodulate analog and digital signals depending on the type of television signal received.

[0064] In some other exemplary embodiments, the tuner 210 may also be in an external device, such as an external set-top box. In this way, the set-top box outputs a television signal after modulation and demodulation, which is then input to the display device 200 via the external device interface 240.

[0065] The communicator 220 is a component used to communicate with external devices or external servers according to various communication protocol types. For example, the display device 200 can send content data to an external device connected via the communicator 220, or browse and download content data from an external device connected via the communicator 220. The communicator 220 may include network communication protocol modules or near-field communication protocol modules such as a WIFI module 221, a Bluetooth communication protocol module 222, and a wired Ethernet communication protocol module 223. Thus, the communicator 220 can receive control signals from the control device 100 according to the control of the controller 250, and realize the control signals as WIFI signals, Bluetooth signals, radio frequency signals, etc.

[0066] Detector 230 is a component of display device 200 used to collect signals from the external environment or to interact with the outside world. Detector 230 may include a sound acquisition unit 231, such as a microphone, which can be used to receive the user's voice, such as voice signals of the user's control commands to the display device 200; or, it can collect ambient sounds to identify the type of environmental scene, enabling the display device 200 to adapt to ambient noise.

[0067] In some other exemplary embodiments, the detector 230 may also include an image acquisition device 232, such as a camera or webcam, which can be used to acquire external environmental scenes to adaptively change the display parameters of the display device 200; and to acquire user attributes or user interaction gestures to realize the function of interaction between the display device and the user.

[0068] In some other exemplary embodiments, the detector 230 may also include a light receiver for acquiring ambient light intensity to adapt to changes in display parameters of the display device 200, etc.

[0069] In some other exemplary embodiments, the detector 230 may also include a temperature sensor, such as by sensing the ambient temperature, so that the display device 200 can adaptively adjust the display color temperature of the image. For example, in a high-temperature environment, the display device 200 may adjust the color temperature of the displayed image to be cooler; in a low-temperature environment, the display device 200 may adjust the color temperature of the displayed image to be warmer.

[0070] External device interface 240 is a component that provides data transmission between the controller 250 and the display device 200 and external devices. External device interface 240 can be connected to external devices such as set-top boxes, gaming devices, and laptops via wired / wireless means, and can receive data such as video signals (e.g., motion pictures), audio signals (e.g., music), and additional information (e.g., EPG) from external devices.

[0071] The external device interface 240 may include one or more of the following: an HDMI terminal 241, a composite video blanking synchronization (CVBS) terminal 242, an analog or digital component terminal 243, a universal serial bus (USB) terminal 244, a component terminal (not shown in the figure), and a red, green and blue (RGB) terminal (not shown in the figure).

[0072] The controller 250 controls the operation of the display device 200 and responds to user operations by running various software control programs (such as operating systems and various applications) stored in the memory 260.

[0073] In some exemplary embodiments, the controller 250 includes random access memory (RAM), read-only memory (ROM), a graphics processor, a CPU processor, a communication interface, and a communication bus. The RAM, ROM, and the communication interfaces of the graphics processor and CPU processor are connected via the communication bus.

[0074] ROM is used to store various system startup instructions. For example, when a power-on signal is received, the display device 200 starts up, the CPU processor 254 executes the system startup instructions in ROM 252, copies the operating system stored in memory 260 to RAM 251, and then starts running the operating system. After the operating system has started, the CPU processor 254 copies various application programs from memory 260 to RAM 251, and then starts running the various application programs.

[0075] The graphics processor 253 is used to generate various graphical objects, such as icons, operation menus, and graphics displayed based on user input commands. The graphics processor 253 may include an arithmetic logic unit (ALU) for performing calculations based on various interactive commands input by the user, and then displaying various objects according to display attributes; and a renderer for generating various objects based on the ALU, and displaying the rendering results on the display 275.

[0076] CPU processor 254 is used to execute operating system and application instructions stored in memory 260, and to process various applications, data, and content according to received user input instructions, so as to ultimately display and play various audio and video content.

[0077] In some exemplary embodiments, the CPU processor 254 may include multiple processors. These multiple processors may include a main processor and multiple or one sub-processors. The main processor is used to perform some initialization operations of the display device 200 in a display device preload mode, and / or to display a screen in normal mode. The multiple or one sub-processors are used to perform an operation in a display device standby mode or other states.

[0078] Communication interface 255 may include a first interface to an nth interface. These interfaces may be network interfaces that are connected to external devices via a network.

[0079] The controller 250 can control the overall operation of the display device 200. For example, in response to receiving a user input command for selecting a GUI object to be displayed on the monitor 275, the controller 250 can perform operations related to the object selected by the user input command.

[0080] The object can be any of the selectable objects, such as a hyperlink or an icon. The operation related to the selected object may include displaying a link to a hyperlinked page, document, image, etc., or executing a program corresponding to the object. The user input command for selecting the GUI object can be a command entered via various input devices connected to the display device 200 (e.g., mouse, keyboard, touchpad, etc.) or a voice command corresponding to spoken speech by the user.

[0081] Memory 260 is used to store various types of data, software programs, or application programs that drive and control the operation of display device 200. Memory 260 may include volatile and / or non-volatile memory. The term "memory" includes memory 260, RAM and ROM of controller 250, or memory cards in display device 200.

[0082] In some embodiments, the memory 260 is specifically used to store the running program of the controller 250 in the display device 200; to store various applications built into the display device 200 and downloaded by the user from external devices; and to store data such as the visual effects images of various GUIs provided by the display 275, various objects related to the GUIs, and selectors for selecting GUI objects.

[0083] In some embodiments, the memory 260 is specifically used to store drivers and related data for the tuner 210, communicator 220, detector 230, external device interface 240, video processor 270, display 275, audio processor 280, etc., such as external data (e.g., audio and video data) received from the external device interface or user data (e.g., key information, voice information, touch information, etc.) received from the user interface.

[0084] In some embodiments, memory 260 specifically stores software and / or programs representing an operating system (OS), which may include, for example, a kernel, middleware, application programming interfaces (APIs), and / or applications. Exemplarily, the kernel may control or manage system resources, as well as the functionality implemented by other programs (such as the middleware, APIs, or applications); simultaneously, the kernel may provide interfaces to allow middleware, APIs, or applications to access the controller to control or manage system resources.

[0085] Figure 3 The diagram illustrates an example of the architecture configuration of the operating system in the memory of the display device 200. The operating system architecture, from top to bottom, consists of the application layer, middleware layer, and kernel layer.

[0086] The application layer includes both built-in system applications and non-system-level applications. It is responsible for direct interaction with the user. The application layer can include multiple applications, such as settings applications, e-post applications, and media center applications. These applications can be implemented as web applications, running on the WebKit engine, and specifically developed and executed using HTML5, Cascading Style Sheets (CSS), and JavaScript.

[0087] Here, HTML, short for Hyper Text Markup Language, is a standard markup language used to create web pages. It uses tags to describe web pages. HTML tags are used to describe text, graphics, animations, sounds, tables, links, etc. The browser reads the HTML document, interprets the content of the tags in the document, and displays it in the form of a web page.

[0088] CSS, short for Cascading Style Sheets, is a computer language used to style HTML documents. It defines style structures such as fonts, colors, and positions. CSS styles can be stored directly in HTML web pages or in separate stylesheet files, allowing for control over the styles displayed on a webpage.

[0089] JavaScript is a language used for web page programming. It can be inserted into HTML pages and interpreted and executed by the browser. The interactive logic of web applications is implemented using JavaScript. JavaScript can communicate with the kernel layer through browsers by encapsulating JavaScript extension interfaces.

[0090] The middleware layer provides standardized interfaces to support operation in various environments and systems. For example, the middleware layer can be implemented as the Multimedia and Hypermedia Information Coding Experts Group (MHEG) middleware related to data broadcasting, as DLNA middleware related to communication with external devices, or as middleware providing the browser environment for applications running within display devices, etc.

[0091] The kernel layer provides core system services, such as file management, memory management, process management, network management, and system security and permission management. The kernel layer can be implemented as a kernel based on various operating systems, such as a kernel based on the Linux operating system.

[0092] The kernel layer also provides communication between system software and hardware, and provides device driver services for various hardware, such as display drivers for monitors, camera drivers for cameras, button drivers for remote controls, WiFi drivers for WiFi modules, audio drivers for audio output interfaces, and power management drivers for power management (PM) modules.

[0093] Figure 2 In this system, the user interface 265 receives various user interactions. Specifically, it is used to send user input signals to the controller 250, or to transmit output signals from the controller 250 to the user. For example, the remote control 100A can send user input signals such as power switch signals, channel selection signals, and volume adjustment signals to the user interface 265, which then forwards them to the controller 250; or, the remote control 100A can receive output signals such as audio, video, or data output from the user interface 265 after processing by the controller 250, and display the received output signals or output the received output signals as audio or vibration.

[0094] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 275, and the user interface 265 receives the user input commands through the GUI. Specifically, the user interface 265 can receive user input commands for controlling the position of a selector within the GUI to select different objects or items. Here, "user interface" refers to the medium interface through which an application or operating system interacts and exchanges information with the user; it realizes the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device, where controls can include visual interface elements such as icons, controls, menus, tabs, text boxes, dialog boxes, status bars, channel bars, and widgets.

[0095] Alternatively, users can input user commands by entering specific sounds or gestures, and the user interface 265 can receive user input commands by recognizing the sounds or gestures through sensors.

[0096] The video processor 270 is used to receive external video signals and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard encoding and decoding protocol of the input signal, so as to obtain a video signal that can be directly displayed or played on the display 275.

[0097] For example, the video processor 270 includes a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, a display formatting module, etc.

[0098] The demultiplexing module is used to demultiplex the input audio and video data streams. For example, if the input is an MPEG-2 stream (a compression standard based on digital storage media for moving images and speech), the demultiplexing module will demultiplex it into video signals and audio signals.

[0099] The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

[0100] Image compositing modules, such as image synthesizers, are used to overlay and blend GUI signals generated by a graphics generator based on user input or its own data with scaled video images to generate displayable image signals.

[0101] The frame rate conversion module is used to convert the frame rate of the input video, such as converting the frame rate of the input 60Hz video to 120Hz or 240Hz. The usual format is achieved by frame interpolation.

[0102] The display formatting module is used to change the signal output by the frame rate conversion module into a signal that conforms to a display format, such as converting the signal output by the frame rate conversion module to output RGB data signals.

[0103] Display 275 is used to receive image signals from video processor 270 and to display video content, images, and a menu control interface. The displayed video content can be from broadcast signals received by tuner 210, or from video input via communicator 220 or external device interface 240. Display 275 also displays the user interface (UI) generated in display device 200 and used to control display device 200.

[0104] Furthermore, the display 275 may include a display screen assembly for presenting an image and a driving assembly for driving the image display. Alternatively, if the display 275 is a projection display, it may also include a projection device and a projection screen.

[0105] The controller can send a control signal to the rotating component 276 to rotate the display 255.

[0106] The audio processor 280 is used to receive external audio signals, and perform audio data processing such as decompression and decoding, noise reduction, digital-to-analog conversion, and amplification according to the standard encoding and decoding protocol of the input signal, so as to obtain an audio signal that can be played in the speaker 286.

[0107] For example, the audio processor 280 can support various audio formats, such as MPEG-2, MPEG-4, Advanced Audio Coding (AAC), and High-Efficiency AAC (HE-AAC).

[0108] The audio output interface 285 is used to receive audio signals output by the audio processor 280 under the control of the controller 250. The audio output interface 285 may include a speaker 286 or an external audio output terminal 287 for outputting to an external device, such as a headphone output terminal.

[0109] In some other exemplary embodiments, the video processor 270 may include one or more chips. The audio processor 280 may also include one or more chips.

[0110] In addition, in some other exemplary embodiments, the video processor 270 and the audio processor 280 may be separate chips or integrated with the controller 250 into one or more chips.

[0111] The power supply 290, under the control of the controller 250, provides power to the display device 200 from an external power source. The power supply 290 can be a built-in power circuit installed inside the display device 200, or it can be an external power source installed outside the display device 200.

[0112] During the screen mirroring process, the mobile terminal 100B can send display screen data to the display device 200 via a wireless connection, such as through the Miracast protocol, forming a screen mirroring video stream. After receiving the screen mirroring video stream, the display device 200 can decode the screen mirroring video stream through the controller 250, parse the frame images, process them, and then send the screen mirroring image to the monitor 275 for display.

[0113] The mobile terminal 100B can be a smart terminal device with display and human-computer interaction functions, such as a mobile phone or tablet computer. Because the mobile terminal 100B has different operating modes, the resulting projected screen also has different layout modes. For example, when a user holds the phone horizontally, the screen displayed on the phone is in a horizontal layout, meaning the width of the screen is greater than its height, and the phone is in landscape mode. Figure 4A As shown. When a user holds the phone vertically, the screen is displayed in a portrait orientation, meaning the width of the screen is less than its height. The phone is in portrait mode, as shown. Figure 4B As shown.

[0114] Different types of mobile terminals 100B have various screen aspect ratios. For example, mobile phones typically have screen aspect ratios of 9:16 or 10:16; tablets have screen aspect ratios of 3:4, etc. Some smart terminal devices, such as smartwatches, may also have a 1:1 screen aspect ratio. For smart terminal devices with a 1:1 screen aspect ratio, the layout of the image is generally the same in both landscape and portrait orientations; the only difference is the orientation when displayed on the device's screen. Therefore, for mobile terminals 100B with a 1:1 screen aspect ratio, the projected image does not distinguish between landscape and portrait orientations.

[0115] In order to enable the display device 200 to automatically rotate the screen according to the portrait / landscape mode of the mobile terminal 100B during screen projection, thereby achieving a better user experience, a screen projection protocol can be configured between the display device 200 and the mobile terminal 100B, such as a screen projection protocol based on the Miracast standard, a screen projection protocol based on the DLNA standard, or a screen projection protocol based on the Airplay standard. The screen projection protocol enables the transmission of the video stream, allowing the screen on the mobile terminal 100B to be projected to the display device 200.

[0116] According to the current Miracast screen mirroring protocol, regardless of whether the mobile terminal 100B is in landscape or portrait mode, the video stream mirrored to the display device 200 is always a 1920×1080 landscape stream. The display device cannot obtain the horizontal and vertical relationship of the screen based on the resolution or aspect ratio of the video stream, which means that the display device 200 cannot automatically rotate the TV screen through the video stream.

[0117] For example, when a mobile terminal interacts with a display device via screen mirroring, if the mobile terminal is in landscape mode, regardless of whether the display device is in landscape or portrait mode, the image information sent to the display device does not include left and right black information. Figure 5A and 5B .

[0118] When the mobile terminal is in portrait mode, regardless of whether the display device interacting with the mobile terminal is in landscape or portrait mode, the image information sent to the display device includes left and right black information, such as... Figure 6A and 6B .

[0119] In the Miracast protocol, when a mobile terminal establishes a communication connection with a display device, the mobile terminal adds a black border to the screenshot before transmitting it to the display device. This is to ensure that the original display device can display the horizontal image regardless of whether the mobile terminal is placed horizontally or vertically in landscape mode.

[0120] exist Figure 6A In the corresponding projected image, the black areas on both sides of the screen represent black information, or black borders. The display area in the center of the screen is called the effective screen (i.e., the screen display corresponding to the mobile terminal), and this effective screen contains the valid information in the image information of the mobile terminal. Within the area corresponding to the effective screen, the operation screen on the mobile terminal 100B is displayed. The black borders have different widths and heights depending on the screen ratio of the mobile terminal 100B display 275.

[0121] That is, when the display device 200 displays the projected image, it uses the direction corresponding to the shorter side as a reference, such as the height direction in landscape mode. Therefore, the height direction of the projected video stream presented by the display device 200 is generally constant. That is, regardless of whether the mobile terminal 100B is placed horizontally or vertically, the height of the projected image received by the display device 200 is always 1080P. Even if the monitor 275 is rotated to portrait mode, it cannot display the same image height. Figure 6C The display state shown is not displayed according to the height of the entire image information, including the left and right black information, such as... Figure 6BAs shown, in portrait mode, not only are the left and right sides of the effective screen filled with black information, but the top and bottom of the projected screen also display black areas because there is no effective screen to fill them, which will greatly affect the user's viewing experience.

[0122] In actual screen mirroring, the video stream received by display device 200 corresponds to a screen display resolution of 1920×1080. This means that in the height direction, the mirrored image is based on the image on the terminal and requires 1080 pixels to display. In the width direction, the mirrored image is based on the terminal's display plus the width of the black borders on both sides, requiring 1920 pixels to display. Consequently, the effective area of ​​the mirrored image has a height of 1080, and the width is scaled proportionally to the height, which significantly wastes display area on monitor 275.

[0123] In order to improve the utilization rate of the display area on the display device and to amplify the effective information by a certain proportion to accommodate the screen-projected video stream sent by the mobile terminal 100B.

[0124] In some implementations, when the mobile terminal is in landscape mode, it presents... Figure 4A In the state shown, during screen mirroring, the video stream sent to the display device, after being parsed, has no left or right black borders in its frame data. When the display device receives image information without left and right black borders, it checks whether the monitor is in landscape mode. When in landscape mode, the controller 250 receives the image information and can directly display it on the monitor. When the monitor is in portrait mode, the controller 250 can send a rotation command to the rotation component 276 to control the rotation component 276 to rotate, so that the monitor rotates to landscape mode.

[0125] When the mobile terminal is in portrait mode, it displays Figure 4B In the state shown, during screen projection, the video stream sent to the display device, after being parsed, contains left and right black borders and the effective image. When the display device receives image information with left and right black borders, it detects whether the display is in portrait mode. When in landscape mode, the controller 250 can send a rotation command to the rotation component 276 to control the rotation component 276, causing the display 275 to rotate to portrait mode. In portrait mode, the width of the display 275 is less than its height, matching the display ratio of the terminal screen. At this time, the frame data in the projected video stream is magnified, and the effective image is displayed on the display. The left and right black areas cannot be displayed on the display due to magnification, thereby reducing the black border area on both sides of the effective image. Figure 6C As shown.

[0126] To improve the user's viewing experience and reduce black information on both sides of the screen, this application provides a display device that can determine whether the display needs to be rotated and the screen adjusted based on whether the image information in the screen-projected data stream contains black information on the left and right sides, so as to maximize the use of the display area.

[0127] This application provides a display device comprising:

[0128] monitor;

[0129] A rotating component is configured to rotate the display so that the display is in a rotation state, either in landscape or portrait mode.

[0130] User interface, wherein the communication information is configured to connect to a terminal;

[0131] And the controller, see Figure 7 Configured to execute:

[0132] S1: Receive the projected video stream.

[0133] In practical applications, users can first perform a screen mirroring operation on the mobile terminal 100B to send the display screen of the mobile terminal 100B to the display device 200. For example, users can select "Settings - Connections and Sharing - Screen Mirroring" on their mobile phones, and then select a display device on the current network as the target device in the screen mirroring device list to perform the screen mirroring operation.

[0134] After the screen mirroring operation is performed, the mobile terminal 100B will send the displayed screen to the display device 200 through a screen mirroring protocol, such as the Miracast protocol or other mirroring protocols. As new interactive screens are continuously generated during the screen mirroring process, the mobile terminal 100B will send the screens to the display device 200 frame by frame, forming a screen mirroring data stream (hereinafter referred to as a screen mirroring video stream).

[0135] It's worth noting that users can also perform screen casting through third-party applications. For example, when a user opens a video application, a screen casting icon is displayed on the video playback screen. The user can click this icon to perform screen casting. Typically, the screen cast via a third-party application will be based on the video resource being played. For instance, when playing a horizontal video resource such as a movie or TV series, the width of the effective screen in the cast will be greater than its height; when playing a vertical video resource such as a short video or comic, the width of the effective screen in the cast will be less than its height.

[0136] S2: Extract the initial resolution from the projected video stream.

[0137] After receiving the projected video stream, the controller 250 of the display device 200 can perform frame-by-frame analysis of the received projected video stream to extract the initial resolution. The initial resolution is the overall resolution of the first frame in the projected video stream.

[0138] For example, if the aspect ratio of the screen-projected video stream sent by the mobile terminal 100B is 1920:1080, after receiving the screen-projected video stream, the controller 250 can obtain each frame by parsing the video stream. Furthermore, it can extract the initial resolution from the first frame (the first frame), which is 1920×1080.

[0139] Obviously, the first frame is not limited to the first frame in the entire projection process; it can also be the first few frames of the projected video stream or the corresponding frame at a specified time point. Since the initial resolution is the overall resolution of the projected image, this resolution information is relatively easy to obtain. In practical applications, the earlier the initial resolution is obtained, the better it is to detect the effective resolution of the projected video stream in a timely manner, thereby adjusting the display to an optimal state as soon as possible.

[0140] S3: Extract the sampling resolution from the projected video stream.

[0141] After extracting the initial resolution, the images in the projected video stream can be further sampled to extract the sampling resolution. The frame used for sampling is called the sampled frame. The sampling resolution is the resolution of the effective area (i.e., the effective frame) on the sampled frame, extracted from the projected video stream at preset time intervals.

[0142] During the sampling process, the preset time interval for sampling can be set according to the computing power of the controller 250. For example, the sampling frame can be a frame 10 seconds after the first frame. To determine the valid frames from the sampling frames, the pixel colors of the sampling frames can be iterated. Obviously, the pixel color value of the black border area is black, and the pixel color value of the valid area is usually not all black. Therefore, by iterating through each pixel of the sampling frame, the black rectangular area can be determined as the black border, and the other areas are the valid areas.

[0143] S4: Compare the sampling resolution with the initial resolution.

[0144] After extracting the initial resolution and sampling resolution of the screen-cast video stream, the sampling resolution can be compared with the initial resolution. Based on the difference between the sampling resolution and the initial resolution, the effective image conditions (such as aspect ratio, orientation, etc.) in the current screen-cast video stream can be determined, and then it can be selected whether to display based on the effective image.

[0145] For example, if the sampling resolution is equal to the initial resolution, that is, the resolution of the first frame is 1920×1080, and the resolution of the effective area determined in the sampled frame is also 1920×1080, then it means that there are no black borders in the current projected image, and the projected image can fill the display area. In other words, the display requirements of the projected image can be met by directly displaying it on the monitor in landscape mode at 275 degrees.

[0146] It should be noted that display resolution is typically expressed as the number of pixels in the width and height dimensions of the screen, such as 1920×1080. However, direct comparison using only the numerical value of resolution is often difficult. For example, numerically, a resolution of 1920×1080 is equal to 1080×1920. Therefore, in actual comparison, a partial value of the resolution can be extracted, or the resolution can be converted to other comparable values ​​before comparison, to obtain a comparison result between the sampled resolution and the initial resolution. For example, the width or height of the entire screen can be extracted from the initial resolution and compared with the height or width of the effective screen extracted from the sampled resolution to determine its effective resolution.

[0147] S5: If the sampling resolution is greater than or equal to the initial resolution, set the effective resolution of the video stream to the initial resolution.

[0148] In practical applications, if the value corresponding to the sampling resolution is greater than or equal to the value corresponding to the initial resolution, it means that the black border area in the current projected image is at its minimum while ensuring complete display. In this case, even if the image is scaled, the black border area will not increase or decrease, confirming that the effective resolution of the video stream has not changed and remains at the initial resolution.

[0149] S6: If the sampling resolution is less than the initial resolution, set the effective resolution of the video stream to the sampling resolution.

[0150] If the value corresponding to the sampling resolution is less than the value corresponding to the initial resolution, it is determined that the current screen projection is filled with a large number of black areas. The display device 200 can improve the display effect by rotating the monitor 275 and scaling the screen projection.

[0151] As can be seen from the above technical solutions, the effective resolution detection method for screen-casting video streams provided in this application can extract the initial resolution and sampling resolution from the screen-casting video stream after receiving it, and compare them to determine the effective resolution of the current video stream. If the sampling resolution is less than the initial resolution, the effective resolution of the video stream is set to the sampling resolution. By setting the effective resolution of the video stream, the screen-casting image can be displayed according to the effective resolution, thereby adapting to the display direction of the screen-casting image, reducing the impact of black borders, and achieving a better user experience.

[0152] In one implementation, to achieve the sampling resolution and the initial resolution, data suitable for comparison can be extracted from both the sampling resolution and the initial resolution, respectively. That is, as... Figure 8 As shown, the step of comparing the sampling resolution with the initial resolution further includes:

[0153] S41: Extract the reference value based on the initial resolution;

[0154] S42: Traverse the valid frames of the sampled images and generate comparison values;

[0155] S43: If the comparison value is greater than or equal to the reference value, determine that the sampling resolution is greater than or equal to the initial resolution;

[0156] S44: If the comparison value is less than the reference value, determine that the sampling resolution is less than the initial resolution.

[0157] After obtaining the initial resolution, some data can be extracted from the initial resolution as a baseline value. For example, when the height of the projected screen remains unchanged in the projected video stream, the overall height of the first frame is used as the baseline value. Then, by traversing the pixel color values ​​in the sampled images, the proportion of the effective image is determined, thereby generating a comparison value. The comparison value is the width of the effective image in the sampled images.

[0158] For example, if the initial resolution of the first frame is 1920×1080, then the baseline value is 1080. Then, by traversing the pixel color values ​​in the sampled image, the effective resolution of the image after removing black areas is determined to be 960×1080, and a contrast value of 960 is generated.

[0159] After determining the baseline value and the comparison value, the relationship between the sampling resolution and the initial resolution can be determined directly by comparing the magnitude of the baseline value and the comparison value. That is, if the comparison value is greater than or equal to the baseline value, the sampling resolution is determined to be greater than or equal to the initial resolution; if the comparison value is less than the baseline value, the sampling resolution is determined to be less than the initial resolution.

[0160] For example, if the baseline value is 1080 and the comparison value is 960, since the comparison value 960 is less than the baseline value 1080, the sampling resolution is determined to be less than the initial resolution. This means the effective resolution of the current projected video stream is set to the sampling resolution. Furthermore, during subsequent display, the projected video stream can be scaled according to the effective resolution.

[0161] In order to calculate the comparison value, in some embodiments of this application, such as Figure 9A As shown, the method further includes:

[0162] S421: Extract black border data by traversing the number of consecutive black pixels in the sampled image;

[0163] S422: Extract initial screen data;

[0164] S423: Calculate the baseline value and the comparison value.

[0165] To calculate the comparison value, the resolution of the effective image in the projected screen needs to be determined. Therefore, as... Figure 9B As shown, the range of a continuous black area can be detected starting from the left side of the image corresponding to the sampled image, and the range of the black border area can be obtained: the width H of the black border on the left side. L The height of the black border on the left is V. L Next, starting from the right side of the image, detect the range of the continuous black area and determine the range of the black border area: the width H of the black border on the right side. R And the height V of the black border on the right R This generates black border data. Simultaneously, initial frame data can be extracted from the first frame. This initial frame data includes the initial width H0 and initial height V0 of the first frame, used to calculate baseline and contrast values.

[0166] The baseline value and the comparison value can be calculated using the following formula:

[0167] The reference value is: S0=V0=V L =V R ;

[0168] The comparison value: S X =H0-(H L +H R ).

[0169] In one implementation, such as Figure 10 As shown, before calculating the benchmark value and the comparison value, the method further includes:

[0170] S4231: Determine whether the sampled screen meets the preset execution conditions.

[0171] S4232: If the sampled image does not meet the preset judgment conditions, set the effective resolution to the initial resolution;

[0172] S4233: If the sampled image meets the preset judgment conditions, perform the step of calculating the reference value and the comparison value.

[0173] In this embodiment, after acquiring the sampled image, the display status of the image in the sampled image can be judged to determine whether it meets the preset execution conditions. The preset execution conditions are: the width H of the left black border. L Equal to the width H of the black border on the rightR And the height of the black border on the left is V L The height V of the black border on the right side R It is equal to the initial height V0. That is, determine whether the sampled image satisfies "H". L =H R And V L =V R =V0".

[0174] If the sampling image does not meet "H" L =H R And V L =V R If the value is "=V0", it indicates that the width and height of the black borders on both sides of the sampled image are unequal. This situation is generally caused by the display content of the mobile terminal 100B affecting the sampling results. Since it is difficult to determine the effective resolution in this case, the original resolution can be maintained when outputting the projected image; that is, the effective resolution is set to the initial resolution, and the screen width H corresponding to the effective resolution is... S =H0, corresponding to the screen height V S =V0.

[0175] If the sampled image meets the "H" L =H R And V L =V R If =V0”, then the effective resolution can be further determined, that is, the steps of calculating the reference value and the comparison value can be performed to compare the sampling resolution with the initial resolution.

[0176] For example, determining whether the sampled image satisfies "H0-(H L +H R If H0-(H) < V0, then H0-(H) is not satisfied. L +H R If V<V0", then the original resolution output is maintained, that is, the effective resolution is set to the initial resolution, and the screen width H corresponding to the effective resolution is... S =H0, corresponding to the screen height V S =V0. If "H0-(H L +H R If V<V0", then the effective resolution is set as the sampling resolution, and the output screen width H corresponding to the effective resolution is... S =H0-(H L +H R The corresponding screen height V S =V0.

[0177] In practical applications, the content projected by the mobile terminal 100B can easily affect the determination of the effective image area in the sampled image. For example, if the display screen of the mobile terminal 100B corresponding to the sampled image is exactly black, if the range is still determined by continuous black pixels at the edge of the projected image, the black image will affect the determination of the black area range, thus affecting the final sampling resolution extraction result. Therefore, to mitigate the impact of black image content on sampling resolution, the step of extracting the sampling resolution from the projected video stream further includes:

[0178] S201: Acquire multiple frames of sampled images at equal time intervals in the projected video stream;

[0179] S202: Calculate the sampling resolution of each frame of the sampled image.

[0180] By setting a preset sampling time interval, multiple samples can be taken from the projected video stream, and the sampling resolution of each sample can be extracted. For example, if one frame is acquired every T time interval, the sampling resolution values ​​obtained by the above resolution algorithm are as follows: S x0 S x1 ... S xn .

[0181] By performing multiple samplings, multiple frames of sampled images can be captured as the displayed screen on the mobile terminal 100B changes. Typically, not all of these multiple sampled frames are affected by the black background content; therefore, capturing multiple frames reduces the impact of screen content on the determination of the black border area, thereby improving the accuracy of effective resolution determination.

[0182] Furthermore, such as Figure 11 As shown, after obtaining the sampling resolutions corresponding to multiple frames, the sampling resolutions can be judged separately to obtain a comparison result between the sampling resolution and the initial resolution. That is, the method further includes:

[0183] S211: Compare the sampling resolution of each frame of the sampled image with the initial resolution;

[0184] S212: If all the sampling resolutions are greater than or equal to the initial resolution, set the effective resolution of the video stream to the initial resolution;

[0185] S213: If all the sampling resolutions are less than the initial resolution, set the effective resolution of the video stream to the sampling resolution.

[0186] By comparing the sampling resolution of each sampled frame with the initial resolution, if consecutive S... x0 ≤S0、S x1≤S0、……、S xn If S ≤ S0, then the effective resolution is S. xn and set S0=S xn If S is continuous x0 ≥S0、S x1 ≥S0、……、S xn If ≥S0, then the effective resolution is S0.

[0187] In one implementation, the method further includes: if the effective resolution of the video stream is set to the sampling resolution, the display 275 of the rotating display device 200 can be controlled to a portrait orientation. For example, the initial resolution is 1920×1080, and the sampling resolution is 960×1080. By calculating the reference value and the comparison value, the width S of the effective frame in the current sampled frame can be determined. X =H0-(H L +H R Since the height of the first frame is 1080, the current effective resolution is the sampling resolution, i.e., the resolution of the effective image: 960×1080. Therefore, the corresponding display image on the mobile terminal 100B is a vertical image of 960×1080.

[0188] Since vertical images are more suitable for display in portrait mode, after determining the effective resolution as the sampling resolution, the controller 250 can send a control command to the rotation component 276 to drive the display 275 to rotate counterclockwise (or clockwise) to the portrait mode.

[0189] After rotating the monitor 275 to portrait mode, the projected image can be displayed in a 960:1080 aspect ratio. However, due to the large screen size of the monitor 275, its typical display resolution is 3840×2160 (landscape mode; portrait mode is 2160×3840). Therefore, in order to display a 960×1080 resolution projected image, the projected image needs to be scaled so that the monitor 275 can fully display the projected image.

[0190] Furthermore, simply resizing the projected image can easily make the displayed image blurry on a large screen, severely degrading the user experience. Therefore, while scaling the projected image, pixel-level quality adjustments can also be made to improve the image quality and enhance the overall display effect.

[0191] As can be seen from the above technical solutions, the effective resolution detection method for screen-casting video streams provided in this application can extract the initial resolution and sampling resolution from the screen-casting video stream after receiving it, and compare them to determine the effective resolution of the current video stream. If the sampling resolution is less than the initial resolution, the effective resolution of the video stream is set to the sampling resolution. By setting the effective resolution of the video stream, the screen-casting image can be displayed according to the effective resolution, thereby adapting to the display direction of the screen-casting image, reducing the impact of black borders, and achieving a better user experience.

[0192] In another exemplary embodiment of this application, another method for processing the projected screen is provided, specifically another display device, including:

[0193] monitor;

[0194] A rotating component is configured to rotate the display so that the display is in a rotation state, either in landscape or portrait mode.

[0195] User interface, wherein the communication information is configured to connect to a terminal;

[0196] And the controller, see Figure 12 Configured to execute:

[0197] S1', receives image information sent by the terminal, wherein when the terminal is in portrait mode, the image information includes valid information and left and right black area information.

[0198] In practical applications, users can first perform a screen mirroring operation on the mobile terminal 100B to send the display screen of the mobile terminal 100B to the display device 200. For example, users can select "Settings - Connections and Sharing - Screen Mirroring" on their mobile phones, and then select a display device on the current network as the target device in the screen mirroring device list to perform the screen mirroring operation.

[0199] After the screen mirroring operation is performed, the mobile terminal 100B will send the displayed screen to the display device 200 through a screen mirroring protocol, such as the Miracast protocol or other screen mirroring and mirroring protocols. As new interactive screens are continuously generated during the screen mirroring process, the mobile terminal 100B will send the screens to the display device 200 frame by frame, forming a screen mirroring video stream.

[0200] S2', if the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to portrait mode;

[0201] In practical applications, the current rotation state of the display can be determined to match the display state of the terminal by analyzing whether the image information contains left and right black information.

[0202] In some exemplary implementations, the controller obtains the rotation angle callback information of the display and determines the target rotation state of the display based on whether the image information obtained from the mobile terminal contains left and right black borders.

[0203] If the image information does not contain left and right black bars, it indicates that the mobile terminal is currently in landscape mode. If the current screen rotation is detected as landscape, the two are matched, and the TV does not need to be rotated. If the current screen rotation is detected as portrait, the two are mismatched, and the screen needs to be rotated to landscape mode.

[0204] If the image information contains black areas on the left and right, it indicates that the mobile terminal is currently in portrait mode. If the current screen rotation is detected as landscape, the two are mismatched, and the screen needs to be rotated to portrait mode. If the current screen rotation is detected as portrait, the two are matched, and the screen does not need to be rotated.

[0205] After a user performs a screen mirroring operation via mobile terminal 100B, mobile terminal 100B will send the mirrored image to display device 200 via a mirroring protocol or a screen mirroring protocol. Controller 250 can receive the image information sent by the terminal and detect the current rotation state of display 275. The detection of the rotation state of display 275 can be accomplished using sensors built into display device 200.

[0206] For example, a gyroscope, a gravity acceleration sensor, or other sensor device can be installed on the display 275 of the display device 200 to determine the attitude data of the display 275 relative to the direction of gravity by measuring angular acceleration or the direction of gravity. The detected attitude data is then compared with the attitude data in landscape and portrait modes to determine the current rotation state of the display 275. Alternatively, a grating angle sensor, a magnetic field angle sensor, or a sliding resistor angle sensor can be installed on the rotation component 276 to determine the current rotation state of the display 275 by measuring the angle rotated by the rotation component 276 and comparing it with the angle in landscape and portrait modes.

[0207] In some exemplary implementations, the controller is further configured to:

[0208] Calculate the rotation direction and rotation angle of the projected image; the rotation direction of the projected image is opposite to the rotation direction of the display; the rotation angle of the projected image is equal to the rotation angle of the display.

[0209] The projected image is rotated according to the stated rotation direction and the stated rotation angle.

[0210] S3', based on the effective information, control the display to display the projected image, wherein the projected image is obtained by magnifying the effective information by a preset factor.

[0211] After receiving the projected video stream, the controller 250 of the display device 200 can perform frame-by-frame analysis of the received projected video stream. For example, if the aspect ratio of the projected video stream sent by the mobile terminal 100B is 1920:1080, the controller 250 can obtain frame images by parsing the projected video stream after receiving it. The extracted frame image resolution is 1920×1080, which is the initial resolution.

[0212] After extracting the resolution, the frame images in the projected video stream can be sampled again to extract the effective resolution. The frame used for sampling is called the sampled frame. The effective resolution is the resolution of the effective frame data extracted from the projected video stream. Specifically, the effective resolution can be obtained from the projected video stream at preset time intervals.

[0213] During the sampling process, in order to determine the valid images from the sampled images, the pixel colors of the sampled images can be traversed. Obviously, the pixel color value of the black area is black, and the pixel color value of the valid area is usually not all black. Therefore, by traversing each pixel of the sampled image, the black rectangular area can be determined as the black border, and the other areas are the valid images.

[0214] It should be noted that, depending on the display method of different display devices 200, the color filled in the black area is not limited to black. For example, to adapt to the overall UI design style of the operating system, the black area can be gray, blue, or other colors, and may also be a gradient, a specific pattern, etc. For the sake of convenience in the following description, this application will still refer to these cases as black areas or black borders.

[0215] After obtaining the frame image resolution and effective resolution for the screen-casting video stream, the effective resolution can be compared with the frame image resolution. Based on the difference between the sampling resolution, effective resolution and frame image resolution, the effective image situation (such as aspect ratio, orientation, etc.) in the current screen-casting video stream can be determined, and then it can be selected whether to display based on the effective image.

[0216] For example, if the effective resolution is equal to the frame image resolution—that is, if the resolution of the first frame is 1920×1080, and the effective area resolution determined in the sampled image is also 1920×1080—then it means that there are no black borders in the current projected image, and the projected image can fill the display area. In other words, displaying the projected image directly on a monitor in landscape mode (275°) will meet the display requirements.

[0217] It should be noted that display resolution is typically expressed as the number of pixels in the width and height dimensions of the screen, such as 1920×1080. However, direct comparison using only the numerical value of resolution is often difficult. For example, numerically, a resolution of 1920×1080 is equal to 1080×1920. Therefore, in actual comparison, a partial value of the resolution can be extracted or the resolution can be converted to other comparable values ​​before comparison to obtain the comparison result between the effective resolution and the frame image resolution. For example, the width or height of the entire screen can be extracted from the frame image resolution and compared with the height or width of the effective screen to determine its effective resolution.

[0218] As can be seen from the above technical solutions, the effective resolution detection method for screen-casting video streams provided in this application can extract the frame image resolution and effective resolution from the screen-casting video stream after receiving it, and compare them to determine the effective resolution of the current video stream. By setting the effective resolution of the video stream, the screen-casting image can be displayed according to the effective resolution, thereby adapting to the display direction of the screen-casting image, reducing the impact of black borders, and achieving a better user experience.

[0219] In order to calculate the magnification of the effective image, in some embodiments of this application, such as... Figure 13 As shown, the method further includes:

[0220] S131: Extract black border data by traversing the number of consecutive black pixels in the sampled image;

[0221] S132: Extract initial screen data;

[0222] S133: Calculate the height ratio R between the display device and the effective screen. H and width ratio R W ;

[0223] S134: Determine R H >R W Size.

[0224] As shown in Figure 9, the width of the image frame transmitted from the mobile terminal is denoted as a or H0, and the height is H. ph Or denoted as V L The width of the display device in portrait mode is W. tv The height is H tv The range of a continuous black area can be detected starting from the left side of the image corresponding to the valid frame, and the range of the black area can be obtained: the width 'a' of the left black border and the height 'H' of the left black border. ph Next, starting from the right side of the image, detect the range of the continuous black area and determine the range of the black area: the width 'a' of the right black border and the height 'H' of the right black border.ph This results in black border data. At this point, the resolution of the effective image in the image information sent by the mobile terminal is (H0-2a)*V. Lh The resolution of the display device is W. tv * H tv .

[0225] The height ratio R of the display device and the effective information H = H tv / H ph Width ratio R W = W0 / (W0-2a), if R H >R W Then execute S85 to amplify the effective information R. W The screen will be projected multiple times; if R H <R W Then execute S86 to amplify the valid information R. H This will give you the projected image.

[0226] As can be seen from the above technical solutions, such as Figure 14 As shown, the display device provided in this application can determine whether the mobile terminal is in portrait mode based on image information and automatically adjust the rotation state of the display, thereby using a larger display space to display the projected image and alleviating the problem that traditional smart TVs cannot display projected images normally.

[0227] Currently, there are many screen casting and mirroring protocols, such as screen casting protocols based on the Miracast standard, screen casting protocols based on the DLNA standard, screen casting protocols based on the AirPlay standard, or customizable screen casting methods.

[0228] Some screen mirroring protocols allow the display device to obtain the actual video stream from the mobile terminal. For example, the AirPlay protocol can send the image data of the mobile terminal's own screen to the display device. In this way, the display device can directly determine whether the image is horizontal or vertical based on the aspect ratio of the image.

[0229] In other screen mirroring protocols, the display device cannot obtain the actual screen video stream of the mobile terminal. That is, the mobile device processes the screen data information before transmitting the screen mirroring data to the display device. For example, in the Miracast standard screen mirroring protocol, the smart device always sends horizontal media resources to the display device. Regardless of whether the screen is placed horizontally or vertically, the display device always obtains horizontal video data.

[0230] For example, when mirroring a vertically oriented mobile terminal, black borders are added to both sides of the screen data before sending it to the display device. This is intended to adapt the screen data to horizontally oriented display devices.

[0231] However, since display devices can rotate horizontally and vertically, the current Miracast standard's screen mirroring protocol has not yet adapted to vertical display devices, and still sends horizontal resources with black borders to the display device.

[0232] In the embodiments shown below, when the display device receives the projection resources, it checks for black borders in the image. Therefore, the display device cannot determine the actual placement of the mobile terminal. In this case, the display device cannot rotate the image based on the valid frame of the image. That is, regardless of whether the phone is in landscape or portrait mode, after receiving horizontal projection data, the projection data is displayed directly.

[0233] Of course, the display device does not rotate based on the projection data. Regardless of whether the display device is in landscape or portrait mode, the user can still actively issue a rotation command to rotate the display device to a different state. From the perspective of viewing the display device from the front, the rotating component 276 can rotate the screen to portrait mode, that is, a state where the vertical side of the screen is longer than the horizontal side, or it can rotate the screen to landscape mode, that is, a state where the horizontal side of the screen is longer than the vertical side.

[0234] For example, if the display device is in landscape orientation, it will display something like this. Figure 5A Or a 6A resolution. For Figure 5A When a mobile device is placed horizontally, the projected data stream is a full-screen horizontal resource without black borders on either side. For Figure 6A When a mobile terminal is placed vertically, the projected data stream consists of horizontal resources in the middle of the data stream, with black borders on both sides. In both of these display placements, the display device receives horizontal vertical screen resources.

[0235] If the display device is in a vertical position, the following will be displayed: Figure 5B Or the 6B screen. For Figure 6B In situations where the effective portion of the screen occupies a small area, to improve the user experience, in the exemplary implementation of this application, control commands can be issued via remote control, voice, gestures, etc., to zoom in on the currently mirrored interface. Correspondingly, a zoom scale is displayed on the user interface of the display device, such as... Figure 15A As shown. After the zoom-in operation is received, the operation bar will automatically disappear within a selectable, preset time. For example, it can be accessed via the remote control using the up, down, left, right, and OK buttons, or the zoom can be manually adjusted to a suitable magnification using the remote control. Figure 15B and 15C As shown.

[0236] In some exemplary implementations, the display device shows the zoom scale only in portrait mode, and does not show the zoom scale in landscape mode.

[0237] In other implementations, during the screen mirroring process, if a user outputs a rotation command to switch from portrait to landscape mode, the display will rotate from portrait to landscape mode and reset the mirrored display screen during ConfigurationChange. After the rotation is completed, a new display effect will be displayed.

[0238] In a specific implementation, this application also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps of the methods provided in this application. When the controller of the display device provided in this application runs the computer program instructions, the controller executes the steps configured for the controller described in this application. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0239] Those skilled in the art will clearly understand that the techniques in the embodiments of this application can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application or some parts of the embodiments.

[0240] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0241] The embodiments described above do not constitute a limitation on the scope of protection of this application.

Claims

1. A display device, characterized in that, include: monitor; A rotating component is configured to rotate the display so that the display is in a rotation state, either in landscape or portrait mode. The user interface is configured to communicate with the terminal via a screen mirroring protocol; The controller is configured as follows: When the display is in landscape mode, the system receives a video stream sent by the terminal based on the Miracast projection protocol. The video stream is landscape in both portrait and landscape modes. Furthermore, when the terminal is in portrait mode, the frames in the video stream include the active frame and left and right black borders, with the active frame corresponding to the content displayed on the terminal's screen. When the terminal is in landscape mode, the frames only contain the active frame and do not include the left and right black borders. If the image information of the projected video stream includes valid information and left and right black information, then the terminal is determined to be in portrait mode. The display is controlled to rotate to portrait mode, and the display is controlled to display a projected image after being magnified by a preset factor based on the valid information, wherein the height of the projected image is consistent with the long side length of the display.

2. The display device according to claim 1, characterized in that, After receiving the projected video stream from the terminal, the controller executes the following: The frame image resolution information in the projected video stream is extracted according to preset rules, and the pixels in the frame image are traversed by color. The pixel color value represents black and the rectangular area is the black border, while other areas are the valid image.

3. The display device according to claim 2, characterized in that, The controller executes control on the display to present the projected image based on the valid information, wherein the projected image is obtained by magnifying the valid information by a preset factor, and is further configured as follows: Calculate the height ratio R between the display device and the effective information. H and width ratio R W , If R H >R W Then the effective information will be amplified by R. W Multiply the display to get the projected image; If R H <R W Then the effective information will be amplified by R. H This will give you the projected image.

4. The display device according to claim 1, characterized in that, After receiving the screen-projected video stream sent by the terminal, the controller is also configured to perform: Extract the initial resolution from the projected video stream, where the initial resolution is the overall resolution of the frame images in the projected video stream; After extracting the initial resolution, the frame images in the projected video stream are sampled again to extract the sampling resolution, wherein the sampling resolution is the resolution of the effective image on the frame image extracted from the projected video stream according to the pre-rules; Compare the sampling resolution with the initial resolution; If the sampling resolution is consistent with the initial resolution, it means that the projected screen does not contain black borders. If the sampling resolution is inconsistent with the initial resolution, it means that the projected screen contains black borders.

5. A display device, characterized in that, include: monitor; A rotating component is configured to rotate the display so that the display is in a rotation state, either in landscape or portrait mode. The user interface is configured to communicate with the terminal via a screen mirroring protocol; The controller is configured as follows: The system receives image information sent by the terminal based on the Miracast screen mirroring protocol, wherein the image information contains at least valid information corresponding to the screen display content of the terminal; wherein, when the terminal is in portrait mode, the frame images in the screen mirroring video stream include valid information and left and right black border information, and the valid information corresponds to the screen display content of the terminal; when the terminal is in landscape mode, the frame images only contain valid information and do not include left and right black border information; The system detects whether the image information also includes left and right black border information according to preset rules. If the image information also includes the left and right black border information, and the display is in landscape mode, then the target rotation state of the display is determined to be portrait mode. The rotating component is controlled to rotate the display to portrait mode, and the display is controlled to display a projected image based on the valid information. The projected image is enlarged based on the image information or valid information and fills the display to the maximum extent according to the aspect ratio of the valid information.

6. The display device according to claim 5, characterized in that, After receiving image information from the terminal, the controller extracts resolution information from the image information to extract the effective resolution from the image information.

7. The display device according to claim 6, characterized in that, The controller extracts resolution information from the image information to extract the effective resolution from the image information, and is further configured to: The pixel colors in the image information are traversed to determine the black rectangular areas as black borders, and the other areas as valid information.

8. The display device according to claim 7, characterized in that, The controller executes control on the display to present the projected image based on the valid information, wherein the projected image is obtained by magnifying the valid information by a preset factor, and is further configured as follows: Calculate the height ratio R between the display device and the effective information. H and width ratio R W , If R H >R W Then the effective information will be amplified by R. W Multiply the display to get the projected image; If R H <R W Then the effective information will be amplified by R. H This will give you the projected image.

9. The display device according to claim 5, characterized in that, After receiving the screen-projected video stream sent by the terminal, the controller is also configured to perform: Extract the initial resolution from the projected video stream, where the initial resolution is the overall resolution of the frame images in the projected video stream; After extracting the initial resolution, the frame images in the projected video stream are sampled again to extract the sampling resolution, wherein the sampling resolution is the resolution of the effective image on the frame image extracted from the projected video stream according to the pre-rules; Compare the sampling resolution with the initial resolution; If the sampling resolution is consistent with the initial resolution, it means that the projected screen does not contain black borders. If the sampling resolution is inconsistent with the initial resolution, it means that the projected screen contains black borders.

10. A method for displaying a projected image on a display device, the display device comprising a display and a rotating component, the rotating component being configured to rotate the display so that the display is in a rotation state between a landscape state and a portrait state; characterized in that, include: When the display is in landscape mode, the receiving terminal sends a video stream based on the Miracast projection protocol. The video stream is landscape in both portrait and landscape modes. Furthermore, when the terminal is in portrait mode, the frames in the video stream include the active frame and left and right black borders, with the active frame corresponding to the content displayed on the terminal's screen. When the terminal is in landscape mode, the frames only contain the active frame and do not include the left and right black borders. If the image information of the projected video stream includes valid information and left and right black information, then the terminal is determined to be in portrait mode. The display is controlled to rotate to portrait mode, and the display is controlled to display a projected image after being magnified by a preset factor based on the valid information, wherein the height of the projected image is consistent with the long side length of the display.

Images