Display apparatus and control method of display apparatus

Abstract

This application relates to the field of display device technology and provides a display device and a control method for the display device. The display device includes a monitor. The method includes: when receiving projection data from a source device, determining whether the data type corresponding to the projection data belongs to a set target data type; if the data type corresponding to the projection data belongs to the target data type, switching to a reverse control mode, in which the interactive operation corresponding to the display device is used to reverse control the display interface of the source device; if the data type corresponding to the projection data does not belong to the target data type, switching to a local control mode, in which the interactive operation corresponding to the display device is used to control the display interface of the monitor. This application can reduce user switching operations and improve the projection control effect.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a control method for the display device. Background Technology

[0002] With the rapid development of technology, wireless screen mirroring has been widely adopted in scenarios such as meetings, education, healthcare, and home entertainment. Wireless screen mirroring can mirror images and audio / video content from small-screen electronic devices such as mobile phones or tablets to display devices such as smart TVs, conference tablets, or projectors in real time, allowing users to enjoy a large-screen sharing experience with high-definition and low-latency audio-visual content.

[0003] To further improve interaction efficiency, existing mainstream screen mirroring protocols already support reverse control functionality. However, in scenarios such as home entertainment, online education, and meetings, users inevitably need to switch back and forth between reverse control and local control. Currently, this usually relies on users manually switching control modes, which is cumbersome, prone to misoperation, and affects the display smoothness in scenarios such as meetings, resulting in poor screen mirroring control performance. Summary of the Invention

[0004] This application provides a display device and a control method for the display device, which can reduce user switching operations and improve the screen projection control effect.

[0005] In a first aspect, some embodiments of this application provide a display device, including: a display;

[0006] The controller connected to the display is configured to:

[0007] When screen projection data is obtained from the source device, determine whether the data type corresponding to the screen projection data belongs to the set target data type;

[0008] If the data type corresponding to the projected data belongs to the target data type, switch to reverse control mode. In reverse control mode, the interactive operation corresponding to the display device is used to reverse control the display interface of the source device.

[0009] If the data type corresponding to the projected data does not belong to the target data type, the system switches to local control mode. In local control mode, the interactive operation corresponding to the display device is used to control the display interface of the monitor.

[0010] In this embodiment, after the controller obtains the projection data, it can first determine whether the data type of the projection data belongs to the target data type. If the data type belongs to the target data type, it directly switches to the reverse control mode. This allows the controller to quickly and accurately control the display interface of the source device when it receives the corresponding interactive operation from the display device, i.e., when the user needs to perform reverse control on the display interface of the source device, without switching control modes. If the data type of the projection data does not belong to the target data type, it can switch to the local control mode. This allows the controller to respond promptly to the interactive operation from the display device during the projection process, controlling the display interface of the monitor without requiring the user to manually switch control modes. This significantly reduces complex user operations and avoids misoperations, while also improving the smoothness of the projection display, i.e., improving the projection control effect.

[0011] In one possible implementation of the first aspect, when the controller performs the task of determining whether the data type corresponding to the projected data belongs to a set target data type, it is configured as follows:

[0012] Obtain the image corresponding to the currently displayed content to obtain the image to be processed;

[0013] Based on the standard feature terms corresponding to the target data type, feature extraction processing is performed on the image to be processed to obtain the target feature value;

[0014] The degree of matching between the target feature value and the standard feature value corresponding to the target data type is obtained, wherein the standard feature value corresponding to the target data type is a benchmark value obtained based on the standard feature item corresponding to the target data type;

[0015] Based on whether the matching degree is greater than or equal to the set matching threshold, it is determined whether the data type corresponding to the screen projection data belongs to the target data type.

[0016] In this embodiment of the application, in order to ensure accurate determination of whether the data type corresponding to the screen projection data belongs to the target data type, the corresponding feature extraction is performed on the image to be processed based on the standard feature items. After obtaining the target feature value, the matching degree between the target feature value and the standard feature value corresponding to the target data type is calculated. Then, it is determined whether the actual matching degree is greater than or equal to the matching threshold. By obtaining the quantified actual matching degree and comparing it with the corresponding matching threshold, it is determined whether the data type corresponding to the screen projection data is the target data type, thereby improving the reliability and interpretability of the judgment.

[0017] In one possible implementation of the first aspect, the standard feature terms corresponding to the target data type include layout features. When the controller performs feature extraction processing on the image to be processed based on the standard feature terms corresponding to the target data type to obtain target feature values, it is configured as follows:

[0018] Line segment detection is performed on the image to be processed to obtain the first layout feature;

[0019] A second layout feature is obtained by performing projection processing on the image to be processed.

[0020] The target feature value is obtained based on the first layout feature and the second layout feature.

[0021] In this embodiment of the application, in order to improve the accuracy of the obtained layout features, the layout features of the image to be processed are obtained by two methods: projection processing and line segment detection processing. Then, based on the layout features obtained by different methods, the final target feature value is obtained. The target feature value is obtained by using different algorithms to ensure the accuracy of the obtained target feature value.

[0022] In one possible implementation of the first aspect, the first layout feature is used to reflect a first rectangular outline present in the image to be processed, and the second layout feature is used to reflect a second rectangular outline present in the image to be processed. When the controller executes the process of obtaining the target feature value based on the first layout feature and the second layout feature, it is configured to:

[0023] Obtain the second rectangle contours corresponding to different first rectangle contours;

[0024] For any first rectangular contour, if the number of line segments in the first rectangular contour is greater than or equal to the number of line segments in the corresponding second rectangular contour, the first rectangular contour is adjusted according to the second rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour.

[0025] When the number of line segments in the first rectangular contour is less than the number of line segments in the corresponding second rectangular contour, the second rectangular contour is adjusted according to the first rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour.

[0026] The target feature values ​​are obtained based on different target rectangular contours.

[0027] In this embodiment, based on the relationship between the number of line segments of different first rectangular contours and the number of line segments of the corresponding second rectangular contours, a rectangular contour with more line segments and higher accuracy is used as the basis, and another rectangular contour with relatively fewer line segments is used to adjust it, thereby obtaining target rectangular contours corresponding to different first rectangular contours, ensuring the accuracy of the obtained target rectangular contours, and then obtaining the final target feature value based on each target rectangular contour, thereby improving the accuracy and reliability of subsequent judgments.

[0028] In one possible implementation of the first aspect, when the controller performs projection processing based on the image to be processed to obtain the second layout feature, it is configured to:

[0029] The image to be processed is preprocessed to obtain a preprocessed image to be processed. The preprocessing includes at least one of grayscale processing, contrast enhancement, and median filtering.

[0030] The Canny operator is used to perform edge detection on the preprocessed image to obtain an edge map.

[0031] The projection process is performed on the edge map, and the second layout feature is obtained based on the projection result.

[0032] In this embodiment, the image to be processed is first preprocessed so that the preprocessed image can more clearly reflect the existing boundaries. Then, the Canny operator is used to perform edge detection on the preprocessed image to obtain an edge map that clearly reflects the existing boundaries. Finally, the edge map is projected to maximize the accuracy and reliability of the obtained second layout features.

[0033] In one possible implementation of the first aspect, when the controller performs the process of obtaining the image corresponding to the currently displayed content and obtaining the image to be processed, it is configured as follows:

[0034] The detection interval duration is obtained based on the refresh rate of the display.

[0035] Based on the detection interval, the image corresponding to the currently displayed content is obtained, and the image to be processed is obtained.

[0036] In this embodiment, the controller can set the detection interval based on the refresh rate of the display, and acquire the image to be processed every detection interval to determine whether the target feature value corresponding to the currently displayed content of the display matches the standard feature value corresponding to the target data type. This determines whether the data type corresponding to the currently displayed projection data belongs to the target data type, and switches the control mode in a timely manner based on the judgment result. This enables timely and accurate response to various interactive operations without requiring users to manually switch frequently, thus improving the projection control effect in scenarios such as meetings.

[0037] In one possible implementation of the first aspect, after the controller performs the switch to reverse control mode, it is further configured to:

[0038] When an interactive operation corresponding to the display device is obtained, the data type of the data to be processed targeted by the interactive operation is obtained according to the coordinate position of the interactive operation in the display interface of the display.

[0039] If the data type of the data to be processed does not belong to the target data type, switch to the local control mode;

[0040] After completing the interaction corresponding to the above interactive operation, switch to the reverse control mode.

[0041] In this embodiment, to ensure accurate response to interactive operations and improve control accuracy during screen projection, after switching to reverse control mode, if an interactive operation is received, the controller can first obtain the data type of the data to be processed corresponding to the interactive operation. If the data type of the data to be processed also belongs to the target data type, the reverse control mode can be maintained. If the data type of the data to be processed does not belong to the target data type, the controller can first switch to local control mode, respond to the interactive operation, complete the interaction corresponding to the interactive operation, and then switch back to reverse control mode. This allows for timely response to interactive operations used for reverse control when needed, thereby improving display smoothness and screen projection control effect.

[0042] Secondly, some embodiments of this application also provide a method for controlling a display device, the display device including a display, the method comprising:

[0043] When screen projection data is obtained from the source device, determine whether the data type corresponding to the screen projection data belongs to the set target data type;

[0044] If the data type corresponding to the projected data belongs to the target data type, switch to reverse control mode. In reverse control mode, the interactive operation corresponding to the display device is used to reverse control the display interface of the source device.

[0045] If the data type corresponding to the projected data does not belong to the target data type, the system switches to local control mode. In local control mode, the interactive operation corresponding to the display device is used to control the display interface of the monitor.

[0046] Thirdly, embodiments of this application provide a display device for a display apparatus, including a module for executing the control method of the display apparatus in the second aspect.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the control method for the display device described in the second aspect.

[0048] Fifthly, embodiments of this application provide a computer program product that, when run on a display device, causes the display device to execute the control method for the display device described in the second aspect above.

[0049] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0052] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the hardware configuration of the control device provided in some embodiments of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a display device provided in some embodiments of this application;

[0055] Figure 5 A timing interaction diagram of a control method for a display device provided in some embodiments of this application;

[0056] Figure 6A flowchart illustrating a control method for a display device provided in some embodiments of this application;

[0057] Figure 7 A flowchart illustrating a control method for a display device provided in some embodiments of this application;

[0058] Figure 8 This is a timing interaction diagram of a control method for a display device provided in some embodiments of this application. Detailed Implementation

[0059] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0060] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0061] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0062] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0063] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0064] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0065] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0066] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0067] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0068] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0069] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0070] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0071] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0072] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0073] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0074] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0075] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0076] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0077] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0078] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0079] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0080] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0081] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0082] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0083] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0084] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0085] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC 133.

[0086] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, button 144, etc.

[0087] In some embodiments, the control device 100 includes at least one of a communication interface 130 and a user input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0088] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0089] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0090] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources in the display device 200. The operating system can (control the display device) provide a user interface, allowing users to interact with the display device 200 and supporting the running of various applications.

[0091] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0092] Figure 4 A schematic diagram of the structure of a display device according to some embodiments of this application is shown. See also Figure 4 The display device 400 includes:

[0093] Display 410 is used to display the user interface.

[0094] The controller 420, which is connected to the aforementioned display 410, is configured to:

[0095] If screen projection data is obtained from the source device, determine whether the data type corresponding to the screen projection data belongs to the set target data type.

[0096] If the data type corresponding to the above-mentioned projection data belongs to the above-mentioned target data type, switch to reverse control mode. In the above-mentioned reverse control mode, the interactive operation corresponding to the above-mentioned display device 400 is used to reverse control the display interface of the above-mentioned source device.

[0097] If the data type corresponding to the above-mentioned projection data does not belong to the above-mentioned target data type, switch to local control mode. In the above-mentioned local control mode, the interactive operation corresponding to the above-mentioned display device 400 is used to control the display interface of the above-mentioned display 410.

[0098] Figure 5 A timing diagram illustrating a control method for a display device according to some embodiments of this application is shown. See also Figure 5 The controller is configured to perform the following steps:

[0099] S501. If screen projection data is obtained from the source device, determine whether the data type corresponding to the above screen projection data belongs to the set target data type.

[0100] As is understandable, the source device usually refers to the device that acts as the data sender during the screen mirroring process, which may include, but is not limited to, devices such as mobile phones, computers, game consoles, and cameras.

[0101] In this embodiment of the application, the display device, as a display end device for data receiving and display, can acquire projection data from the source end device and display the acquired projection data.

[0102] It is understandable that the target data type can be determined based on user settings or input, or it can be determined by the controller based on actual application requirements.

[0103] For example, the controller can obtain control mode switching data (reflecting the switching status of control modes) within a past target time period (such as the past month), analyze the types of data that need to be frequently reversed based on the control mode switching data, and then obtain the target data type based on the types of historical data that need to be frequently reversed.

[0104] In this embodiment, the target data type may include a coarse-grained document type, or a finer-grained presentation type within the document type. In other embodiments, the target data type may also include other data types such as table types within the document type, or data types such as coarse-grained audio and video types, depending on the actual application requirements.

[0105] S502. If the data type corresponding to the above-mentioned projection data belongs to the above-mentioned target data type, switch to reverse control mode. In the above-mentioned reverse control mode, the interactive operation corresponding to the above-mentioned display device is used to reverse control the display interface of the above-mentioned source device.

[0106] Interactive operations for display devices may include, but are not limited to, touch operations, mouse operations, keyboard operations, gesture sensing operations, voice control operations, and laser pointer operations.

[0107] This application uses the example of a user inputting interactive operations through a control device for illustration. In other embodiments, the control operations corresponding to the display device can also be input by the user through screen touch, gestures, or voice, etc., which can be set according to actual application requirements.

[0108] like Figure 5 As shown, in reverse control mode, the controller can transmit the interactive operations corresponding to the display device back to the source device in real time, so that the source device can drive the focus movement, cursor positioning, content scrolling, text input or function triggering of its display interface according to the interactive operations, thereby realizing the control of the display interface of the source device.

[0109] In this embodiment, the controller can first convert the interactive operation corresponding to the display device into a response event of the source device, and then send it back to the source device. This desensitizes the returned data, improving the security of the display device while reducing the response latency of the source device. In other embodiments, the controller can also directly send the interactive operation corresponding to the display device back to the source device, which then parses it into a response event, thereby reducing the computing power and power consumption of the display device.

[0110] In this embodiment of the application, when the data type corresponding to the screen projection data belongs to the target data type, the controller can automatically switch to the reverse control mode, so that the user does not need to manually switch the control mode during the screen projection process, reducing complex user operations and lowering the probability of misoperation.

[0111] S503. If the data type corresponding to the above-mentioned projection data does not belong to the above-mentioned target data type, switch to local control mode. In the above-mentioned local control mode, the interactive operation corresponding to the above-mentioned display device is used to control the display interface of the above-mentioned display.

[0112] See Figure 5 In local control mode, the controller can generate corresponding control commands based on the interactive operation of the display device and send them to the display, so that the display can operate according to the control commands, thereby driving the display interface to move focus, position cursor, scroll content, input text or trigger functions, thus realizing the control of the display interface.

[0113] In this embodiment, after the controller obtains the projection data, it can first determine whether the data type of the projection data belongs to the target data type. If the data type belongs to the target data type, it directly switches to the reverse control mode. This allows the controller to quickly and accurately control the display interface of the source device when it receives the corresponding interactive operation from the display device, i.e., when the user needs to perform reverse control on the display interface of the source device, without switching control modes. If the data type of the projection data does not belong to the target data type, it can switch to the local control mode. This allows the controller to respond promptly to the interactive operation from the display device during the projection process, controlling the display interface of the monitor without requiring the user to manually switch control modes. This significantly reduces complex user operations and avoids misoperations, while also improving the smoothness of the projection display, i.e., improving the projection control effect.

[0114] In some embodiments, when the controller determines whether the data type corresponding to the above-mentioned projection data belongs to the set target data type, it is configured to perform the following steps:

[0115] Obtain the image corresponding to the currently displayed content to obtain the image to be processed.

[0116] Based on the standard feature terms corresponding to the above target data type, feature extraction processing is performed on the above image to be processed to obtain the target feature value.

[0117] The degree of matching between the target feature value and the standard feature value corresponding to the target data type is obtained. The standard feature value corresponding to the target data type is a benchmark value obtained based on the standard feature item corresponding to the target data type.

[0118] Based on whether the matching degree is greater than or equal to the set matching threshold, it is determined whether the data type corresponding to the above-mentioned screen projection data belongs to the above-mentioned target data type.

[0119] In this embodiment, the controller can quickly and accurately obtain the image corresponding to the currently displayed content of the display as the image to be processed by taking a screenshot. In other embodiments, the controller can also copy the current frame from the buffer as the image to be processed through a corresponding interface, which can be determined according to the actual application requirements.

[0120] It is understandable that the standard features corresponding to a target data type can usually be determined based on the characteristics of that data type. For example, for a presentation (also known as a slideshow), the standard features can be determined based on the presentation's characteristics, such as its paginated canvas, editable vector shapes (e.g., text boxes and paths) mixed with bitmaps (e.g., image or video thumbnails).

[0121] It can be understood that the standard feature value corresponding to the target data type refers to the benchmark value obtained based on the standard feature items corresponding to the target data type. This benchmark value can be determined by user settings or input, or it can be obtained by feature extraction processing of the standard image corresponding to the target data type (i.e., the image obtained based on the display content of the target data type) based on the standard feature items corresponding to the target data type. The specific settings can be set according to the type of standard feature items or other requirements.

[0122] For example, assuming the target data type includes document types, and the standard feature term corresponding to the document type includes text density features, then the screenshot corresponding to the displayed content of the document type can be obtained as a standard image. Feature extraction processing is performed on the standard image to obtain the text density corresponding to the standard image as the standard feature value of the text density feature.

[0123] It is understandable that the matching threshold can be determined based on user settings or input, or it can be determined by the controller based on information such as accuracy requirements.

[0124] It is understood that when the controller obtains the degree of matching between the target feature value and the standard feature value corresponding to the target data type, it can calculate the similarity between the target feature value and the standard feature value as its degree of matching through a similarity algorithm such as cosine similarity, or it can calculate the difference between the value of the target feature value and the standard feature value, and map the difference to a preset function (such as an exponential decay function) to obtain the required degree of matching. The embodiments of this application do not impose specific restrictions on the specific implementation method of obtaining the degree of matching between the target feature value and the standard feature value.

[0125] It is understandable that the controller can usually determine that the data type corresponding to the screen projection data belongs to the above target data type when the matching degree is greater than or equal to the set matching threshold (such as 0.9); and determine that the data type corresponding to the screen projection data does not belong to the above target data type when the matching degree is less than the matching threshold.

[0126] In this embodiment of the application, in order to ensure accurate determination of whether the data type corresponding to the screen projection data belongs to the target data type, the corresponding feature extraction is performed on the image to be processed based on the standard feature items. After obtaining the target feature value, the matching degree between the target feature value and the standard feature value corresponding to the target data type is calculated. Then, it is determined whether the actual matching degree is greater than or equal to the matching threshold. By obtaining the quantified actual matching degree and comparing it with the corresponding matching threshold, it is determined whether the data type corresponding to the screen projection data is the target data type, thereby improving the reliability and interpretability of the judgment.

[0127] Figure 6 A flowchart illustrating a control method for a display device according to some embodiments is shown. See also Figure 6The standard feature terms corresponding to the above target data type include layout features. When the controller performs feature extraction processing on the above image to be processed based on the standard feature terms corresponding to the above target data type and obtains the target feature value, it is configured to perform the following steps:

[0128] Based on the above image to be processed, line segment detection is performed to obtain the first layout feature.

[0129] Based on the above image to be processed, projection processing is performed to obtain the second layout feature.

[0130] The target feature value is obtained based on the first layout feature and the second layout feature.

[0131] In this embodiment of the application, since the target data type is a document-related data type, and various document types such as presentations and tables have significant differences in layout compared to other data types, layout features can be obtained as standard feature items corresponding to the target data type.

[0132] Line segment detection processing is typically used to detect line segments present in an image.

[0133] Projection processing typically refers to the method of accumulating pixel values ​​line by line along a certain direction (horizontal, vertical, or arbitrary angle) of a two-dimensional matrix to obtain a one-dimensional histogram, and using the peak positions to infer the coordinates of the boundaries.

[0134] In this embodiment, the controller can use Hough transform to perform line segment detection on the image to be processed, thereby obtaining the first layout feature. Hough transform can map local feature points (such as edge pixels) in the image space to the parameter space through a voting mechanism, thereby detecting line segments. There is no need to merge edges or fit them in advance. As long as there are enough points, the line segments can be detected in the parameter space. It also has good resistance to gaps, noise, and local occlusion, which can improve the accuracy and reliability of the obtained first layout feature.

[0135] When the controller performs line segment detection on the image to be processed, it can dynamically determine the line segment threshold for detection based on the size of the image (e.g., the line segment threshold is one-third of the length or width of the image to be processed) to ensure the reliability of the detected line segments. Optionally, after acquiring the first layout feature, the controller can also classify the line segments in the first layout feature into horizontal and vertical line segments according to a set pixel tolerance (e.g., 5px), thereby avoiding the situation where the boundary is blurred due to factors such as screen projection, and misidentification as a rectangle.

[0136] It is understood that in other embodiments, the controller may also perform line segment detection processing on the image to be processed through deep learning or local chaining, thereby obtaining the first layout features. The specific settings can be configured according to the actual application requirements.

[0137] In this embodiment, the controller can first perform binarization processing on the image to be processed to obtain a binarized image, thereby maximizing the boundary transition amplitude. Then, the binarized image can be projected to quickly and accurately obtain the second layout features, improving the efficiency and accuracy of the projection processing.

[0138] Understandably, during binarization, the controller can divide pixels into target pixels or background pixels based on whether their pixel values ​​are greater than or equal to a set threshold, setting the pixel value of target pixels to 1 and the pixel value of background pixels to 0. The threshold can be determined by user settings or input, or by the controller based on information such as the pixel value distribution of the image to be processed.

[0139] It is understood that in the embodiments of this application, the standard feature items corresponding to the target data type include layout features. The target feature value obtained based on the first layout feature and the second layout feature corresponding to the image to be processed is the layout feature of the currently displayed content reflected by the image to be processed.

[0140] Optionally, when the controller obtains the target feature value based on the first layout feature and the second layout feature, it may determine the target feature value based on the relationship between the confidence levels of the first layout feature and the second layout feature, according to the layout feature with the greater confidence level. Alternatively, it may first obtain the intersection feature and the union feature based on the first layout feature and the second layout feature, and then process the intersection feature and the union feature through interpolation or other methods to obtain the required target feature value. This application embodiment does not impose specific limitations on this.

[0141] In this embodiment of the application, in order to improve the accuracy of the obtained layout features, the layout features of the image to be processed are obtained by two methods: projection processing and line segment detection processing. Then, based on the layout features obtained by different methods, the final target feature value is obtained. The target feature value is obtained by using different algorithms to ensure the accuracy of the obtained target feature value.

[0142] In some embodiments, the controller can also obtain text density feature values ​​based on the image to be processed, and verify the data type corresponding to the projected data through the text density feature values, thereby further improving the accuracy of the judgment.

[0143] For example, the controller obtains the target feature value based on the first layout feature and the second layout feature. After determining whether the data type corresponding to the screen projection data belongs to the target data type based on the degree of matching between the target feature value and the standard feature value, it can determine the confidence level of the judgment result (i.e., the judgment result of whether the data type corresponding to the screen projection data belongs to the target data type) based on the degree of difference between the text density feature value and the text density standard feature value corresponding to the target data type. Then, it determines whether the judgment result is reliable based on whether the confidence level is greater than or equal to the confidence level threshold (e.g., 0.85). When the judgment result is determined to be reliable, the control mode is switched based on the judgment result.

[0144] Figure 7 A flowchart illustrating a control method for a display device according to some embodiments is shown. See also Figure 7 The first layout feature is used to reflect a first rectangular outline present in the image to be processed, and the second layout feature is used to reflect a second rectangular outline present in the image to be processed. When the controller obtains the target feature value based on the first layout feature and the second layout feature, it is configured to perform the following steps:

[0145] Obtain the second rectangle contours corresponding to different first rectangle contours.

[0146] For any of the aforementioned first rectangular contours, if the number of line segments in the first rectangular contour is greater than or equal to the number of line segments in the corresponding second rectangular contour, the first rectangular contour is adjusted according to the second rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour.

[0147] When the number of line segments in the first rectangular contour is less than the number of line segments in the corresponding second rectangular contour, the second rectangular contour is adjusted according to the first rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour.

[0148] The target feature values ​​are obtained based on different target rectangular contours.

[0149] It should be noted that in this embodiment, the controller can obtain rectangular features as layout features. However, due to factors such as noise or distortion, the layout features obtained by the controller may not be closed rectangles, but rather the outline of an approximate rectangle. Therefore, when the controller obtains the first layout feature and the second layout feature, it can analyze the existing rectangles based on the outline of the approximate rectangle to obtain the first layout feature that reflects the outline of the first rectangle and the second layout feature that reflects the outline of the second rectangle.

[0150] In this embodiment of the application, the first layout feature and the second layout feature can be used for complementary processing, thereby improving the accuracy and reliability of the obtained target feature values.

[0151] In this process, the second rectangular contours corresponding to different first rectangular contours in the first layout feature can be determined first. Then, for any first rectangular contour, based on the relationship between the number of its line segments and the number of line segments of the corresponding second rectangular contour, one of the rectangular contours can be determined as the basic rectangular contour. Then, another rectangular contour can be used to adjust the basic rectangular contour to obtain the final target rectangular contour.

[0152] For example, assuming the first rectangular contour A1 has 4 line segments and the corresponding second rectangular contour B2 has 3 line segments, since the number of line segments in the first rectangular contour A1 is greater than the number of line segments in the second rectangular contour B2, the first rectangular contour A1 can be used as the base rectangular contour. Then, the second rectangular contour B2 can be used to adjust the first rectangular contour A1. This adjustment can include, but is not limited to, adjusting parameters such as line segment length and angle. For example, the line segment lengths of each line segment in the second rectangular contour B2 can be interpolated with the corresponding line segments in the first rectangular contour A1 to obtain the target rectangular contour corresponding to the first rectangular contour A1.

[0153] In this embodiment, based on the relationship between the number of line segments of different first rectangular contours and the number of line segments of the corresponding second rectangular contours, a rectangular contour with more line segments and higher accuracy is used as the basis, and another rectangular contour with relatively fewer line segments is used to adjust it, thereby obtaining target rectangular contours corresponding to different first rectangular contours, ensuring the accuracy of the obtained target rectangular contours, and then obtaining the final target feature value based on each target rectangular contour, thereby improving the accuracy and reliability of subsequent judgments.

[0154] In other embodiments, since line segment detection processing usually has good accuracy, in order to ensure the accuracy of the target feature value while reducing the performance pressure on the controller, and to balance accuracy and computing power, the controller can also directly use the first layout feature obtained by line segment detection processing as the basis and use the second layout feature to adjust the first layout feature. That is, the corresponding first rectangle contour in the first layout feature is adjusted by using different second rectangle contours in the second layout feature, and the adjusted first layout feature is used as the target feature value.

[0155] In some embodiments, when the controller performs projection processing based on the image to be processed to obtain a second layout feature, it is configured to perform the following steps:

[0156] The image to be processed is preprocessed to obtain a preprocessed image to be processed. The preprocessing includes at least one of grayscale processing, contrast enhancement, and median filtering.

[0157] The Canny operator is used to perform edge detection on the preprocessed image to obtain an edge map.

[0158] The above projection process is applied to the edge map, and the second layout feature is obtained based on the projection result.

[0159] Grayscale processing is typically used to convert color or multi-band images into grayscale images where each pixel uses only one channel and one value to represent its brightness, according to a specified weighting formula.

[0160] Contrast enhancement is typically used to stretch or remap the grayscale dynamic range of an image, making dark areas darker and bright areas brighter, thereby increasing the gradient magnitude of edges or textures and providing more pronounced peaks and valleys for subsequent processing.

[0161] Median filtering typically refers to a non-linear smoothing method that uses the median of all gray values ​​in a pixel's neighborhood, sorted together, to replace the original value of that pixel. It has a strong ability to remove black and white noise and can preserve the true edges without blurring them.

[0162] The Canny operator is a multi-level optimized edge detection algorithm that aims to extract edges in an image as accurately as possible while suppressing noise.

[0163] It is understandable that when using the Canny operator to perform edge detection on a preprocessed image, the process typically includes Gaussian smoothing, gradient calculation, non-maximum suppression, and double thresholding to obtain a binary edge map. A binary edge map is usually a binary image composed of 0s and 1s, where pixels belonging to the edge typically have a value of 1, and other pixels have a value of 0.

[0164] In this embodiment, the controller can determine a high threshold (e.g., 150px) and a low threshold (e.g., 50px) for the Canny operator based on the current projection scenario (e.g., a meeting scenario) for dual-threshold processing. In other embodiments, the controller can also determine the high and low thresholds by combining multiple information such as the current projection scenario and the data type corresponding to the projection data (e.g., a presentation). Through adaptive dual thresholds, hidden rectangular boundaries in the current projection content can be better detected, thereby improving the accuracy of the subsequently acquired second layout features.

[0165] In this embodiment, when the controller projects the edge map, it can first quantize the edge map into an intensity map reflecting the boundary intensity distribution, and then perform average projection of the intensity map in the row and column directions to obtain one-dimensional curves corresponding to the rows and columns. It can be understood that the value of each point on the one-dimensional curve reflects the average strength of the edge on the corresponding row or column, and the peak value usually corresponds to the position of the horizontal or vertical boundary. Therefore, by combining the two one-dimensional curves with the set projection threshold, the existing boundaries can be searched, thereby obtaining the second layout feature.

[0166] Optionally, the projection threshold can be dynamically determined by the controller based on the edge map or a one-dimensional curve. For example, for one-dimensional curves in the column and row directions, the controller can calculate the projection threshold based on the values ​​of the N points with the smallest values ​​in the one-dimensional curve, where N is determined based on the number of data points in the one-dimensional curve and a set ratio (e.g., 0.1).

[0167] Optionally, when searching a one-dimensional curve in conjunction with a projection threshold, a bidirectional search can be performed to ensure the integrity of the searched boundary by searching from both the front end and the back end.

[0168] In other embodiments, the edge map can be directly projected to obtain the second layout feature.

[0169] In this embodiment, the image to be processed is first preprocessed so that the preprocessed image can more clearly reflect the existing boundaries. Then, the Canny operator is used to perform edge detection on the preprocessed image to obtain an edge map that clearly reflects the existing boundaries. Finally, the edge map is projected to maximize the accuracy and reliability of the obtained second layout features.

[0170] In some embodiments, when the controller obtains the image corresponding to the currently displayed content and obtains the image to be processed, it is configured to perform the following steps:

[0171] Based on the refresh rate of the aforementioned display, the detection interval duration is obtained;

[0172] Based on the above detection interval, the image corresponding to the currently displayed content is obtained, resulting in the above image to be processed.

[0173] As we can understand it, the refresh rate of a monitor is the number of times the monitor updates the user interface per second. For example, assuming the monitor's refresh rate is 60Hz, meaning the monitor updates the user interface 60 times per second, and the refresh interval is 16.7 milliseconds, then the controller can set the detection interval to 167 milliseconds based on this refresh rate.

[0174] It is understood that in other embodiments, the detection interval duration may be determined based on user settings or input, or it may be determined by the controller based on multiple factors such as the refresh rate of the display, the computing power of the display device, and latency requirements.

[0175] In this embodiment, the controller can set the detection interval based on the refresh rate of the display, and acquire the image to be processed every detection interval to determine whether the target feature value corresponding to the currently displayed content of the display matches the standard feature value corresponding to the target data type. This determines whether the data type corresponding to the currently displayed projection data belongs to the target data type, and switches the control mode in a timely manner based on the judgment result. This enables timely and accurate response to various interactive operations without requiring users to manually switch frequently, thus improving the projection control effect in scenarios such as meetings.

[0176] Figure 8 A timing interaction diagram of a control method for a display device provided in some embodiments is shown. See also Figure 8 After the controller switches to reverse control mode, it is also configured to perform the following steps:

[0177] When the interactive operation corresponding to the above-mentioned display device is obtained, the data type of the data to be processed for the interactive operation is obtained according to the coordinate position of the interactive operation in the display interface of the above-mentioned display.

[0178] If the data type of the data to be processed does not belong to the target data type, switch to the local control mode.

[0179] After completing the interaction corresponding to the above interactive operation, switch to the above reverse control mode.

[0180] The coordinates of an interactive operation on the display screen usually refer to the two-dimensional pixel coordinates on the display screen where the interactive operation falls, such as the coordinates of the two-dimensional pixel clicked by a mouse click.

[0181] It is understandable that the data to be processed in an interactive operation usually refers to the data that the user wants to modify, move, delete, or submit for corresponding business processing through the interactive operation.

[0182] For example, suppose the obtained interactive operation is: a touch operation of the stylus on the monitor screen, and the coordinate position of the touch operation on the monitor's display interface is the coordinate position corresponding to the next page arrow of the presentation. Then, this interactive operation is used to instruct the number corresponding to the page object identifier of the presentation to be incremented by 1 in order to switch the display of the next page of the presentation. In this case, the data to be processed by this interactive operation can be considered as the presentation, and the data type of the data to be processed can be a presentation or a document.

[0183] It is understandable that during the screen mirroring process, the currently displayed data may include data of different data types. That is, the currently displayed content may simultaneously include data of the target data type and data of non-target data types. When the controller obtains and displays the screen mirroring data, if it determines that the screen mirroring data includes data of the target data type, it can switch to reverse control mode.

[0184] In this embodiment, to ensure accurate response to interactive operations and improve control accuracy during screen projection, after switching to reverse control mode, if an interactive operation is received, the controller can first obtain the data type of the data to be processed corresponding to the interactive operation. If the data type of the data to be processed also belongs to the target data type, the reverse control mode can be maintained. If the data type of the data to be processed does not belong to the target data type, the controller can first switch to local control mode, respond to the interactive operation, complete the interaction corresponding to the interactive operation, and then switch back to reverse control mode. This allows for timely response to interactive operations used for reverse control when needed, thereby improving display smoothness and screen projection control effect.

[0185] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0186] This application provides a computer program product that, when run on a display device, enables the display device to implement the steps described in the various method embodiments.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / display device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0188] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0189] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A display device, characterized in that, include: monitor; The controller connected to the display is configured to: When screen projection data is obtained from the source device, determine whether the data type corresponding to the screen projection data belongs to the set target data type; If the data type corresponding to the projected data belongs to the target data type, switch to reverse control mode. In reverse control mode, the interactive operation corresponding to the display device is used to reverse control the display interface of the source device. If the data type corresponding to the projected data does not belong to the target data type, the system switches to local control mode. In local control mode, the interactive operation corresponding to the display device is used to control the display interface of the monitor.

2. The display device as claimed in claim 1, characterized in that, When the controller performs the function of determining whether the data type corresponding to the projected data belongs to the set target data type, it is configured as follows: Obtain the image corresponding to the currently displayed content to obtain the image to be processed; Based on the standard feature terms corresponding to the target data type, feature extraction processing is performed on the image to be processed to obtain the target feature value; The degree of matching between the target feature value and the standard feature value corresponding to the target data type is obtained, wherein the standard feature value corresponding to the target data type is a benchmark value obtained based on the standard feature item corresponding to the target data type; Based on whether the matching degree is greater than or equal to the set matching threshold, it is determined whether the data type corresponding to the screen projection data belongs to the target data type.

3. The display device as described in claim 2, characterized in that, The standard feature terms corresponding to the target data type include layout features. When the controller performs feature extraction processing on the image to be processed based on the standard feature terms corresponding to the target data type to obtain the target feature value, it is configured as follows: Line segment detection is performed on the image to be processed to obtain the first layout feature; A second layout feature is obtained by performing projection processing on the image to be processed. The target feature value is obtained based on the first layout feature and the second layout feature.

4. The display device as described in claim 3, characterized in that, The first layout feature is used to reflect a first rectangular outline present in the image to be processed, and the second layout feature is used to reflect a second rectangular outline present in the image to be processed. When the controller executes the process of obtaining the target feature value based on the first layout feature and the second layout feature, it is configured as follows: Obtain the second rectangle contours corresponding to different first rectangle contours; For any first rectangular contour, if the number of line segments in the first rectangular contour is greater than or equal to the number of line segments in the corresponding second rectangular contour, the first rectangular contour is adjusted according to the second rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour. When the number of line segments in the first rectangular contour is less than the number of line segments in the corresponding second rectangular contour, the second rectangular contour is adjusted according to the first rectangular contour to obtain the target rectangular contour corresponding to the first rectangular contour. The target feature values ​​are obtained based on different target rectangular contours.

5. The display device as described in claim 3, characterized in that, When the controller performs projection processing based on the image to be processed to obtain the second layout feature, it is configured as follows: The image to be processed is preprocessed to obtain a preprocessed image to be processed. The preprocessing includes at least one of grayscale processing, contrast enhancement, and median filtering. The Canny operator is used to perform edge detection on the preprocessed image to obtain an edge map. The projection process is performed on the edge map, and the second layout feature is obtained based on the projection result.

6. The display device as claimed in claim 2, characterized in that, When the controller executes the process of obtaining the image corresponding to the currently displayed content and obtains the image to be processed, it is configured as follows: The detection interval duration is obtained based on the refresh rate of the display. Based on the detection interval, the image corresponding to the currently displayed content is obtained, and the image to be processed is obtained.

7. The display device according to any one of claims 1 to 6, characterized in that, After the controller switches to reverse control mode, it is also configured to: When an interactive operation corresponding to the display device is obtained, the data type of the data to be processed targeted by the interactive operation is obtained according to the coordinate position of the interactive operation in the display interface of the display. If the data type of the data to be processed does not belong to the target data type, switch to the local control mode; After completing the interaction corresponding to the above interactive operation, switch to the reverse control mode.

8. A control method for a display device, characterized in that, The display device includes a display, and the method includes: When screen projection data is obtained from the source device, determine whether the data type corresponding to the screen projection data belongs to the set target data type; If the data type corresponding to the projected data belongs to the target data type, switch to reverse control mode. In reverse control mode, the interactive operation corresponding to the display device is used to reverse control the display interface of the source device. If the data type corresponding to the projected data does not belong to the target data type, the system switches to local control mode. In local control mode, the interactive operation corresponding to the display device is used to control the display interface of the monitor.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 8.

Images