Display device and multi-screen synchronous touch display method

Abstract

The embodiment of the invention provides a display device and a multi-screen synchronous touch display method, and the method can generate target bitmap data of a first touch track in response to the first touch track input by a user. Wherein the first touch track is a touch track of the drawn event. Detecting a first touch point with the latest touch time in the first touch track, and calculating a drawing area according to the first touch point; and refreshing the target bitmap data to an acceleration layer and a graphic layer of the first display according to the drawing area, and transmitting the target bitmap data to a graphic layer of the second display according to the drawing area. According to the method, bitmap data of a touch track can be synchronously refreshed to an acceleration layer and a graphic layer of a first display according to a touch point of the touch track, and meanwhile, the bitmap data is transmitted to a graphic layer of a second display, so that the first display and the second display display the same picture content; therefore, the problem that display pictures of multiple displays cannot be synchronized is solved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a multi-screen synchronous touch display method. Background Art

[0002] Display devices refer to terminal devices that can output specific display images, and can be terminal devices such as smart TVs, communication terminals, smart advertising screens, projectors, etc. Taking smart TVs as an example, smart TVs are based on Internet application technology, have open operating systems and chips, have open application platforms, can realize two-way human-computer interaction functions, and are TV products that integrate multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users. Display devices can be equipped with device interfaces, and monitors can be connected through the device interfaces to form a multi-screen display mode.

[0003] In multi-screen display mode, the display device can send the user interface displayed on the main screen display to multiple secondary screen displays for display. The content of the user interface in the main screen and the secondary screen display can be the same or different, that is, in the same display mode, the user interface content displayed by multiple displays is the same, and in the different display mode, the content displayed by multiple displays is different. For display devices that support touch interaction operations, you can also run a whiteboard application and display the touch track in the user interface through the whiteboard application. When the touch track is detected, the touch track is refreshed to the OSD layer (OSD plane), and the display screen of the OSD layer is presented through the main screen display and the secondary screen display. In order to improve the touch interaction experience, an acceleration layer with a higher level than the OSD layer is also provided in the main screen display. The touch track can be refreshed to the acceleration layer in real time to achieve a hand-following effect.

[0004] However, since there is no acceleration layer in the secondary screen display, the display effect of the acceleration layer cannot be presented in the user interface displayed on the secondary screen. In this way, in the same display mode, if the user inputs a touch track through the whiteboard application of the main screen display, the secondary screen display cannot display the touch track in real time, and can only display the touch track after detecting the lift event of the touch track, resulting in the display screen of the main screen display and the secondary screen display being out of sync, reducing the user experience. Summary of the invention

[0005] The present application provides a display device and a multi-screen synchronous touch display method to solve the problem that the display contents of a main screen display and a secondary screen display cannot be synchronized.

[0006] In a first aspect, some embodiments of the present application provide a display device, including a first display, a device interface and a controller. The first display is configured to display a drawing interface, the drawing interface includes a display screen of a graphics layer, the first display is connected to a touch interaction module, the touch interaction module is used to detect a touch track of a user input, and when a touch track of a user input drawing event is detected, the touch track of the drawing time is also displayed through an acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released, and the display level of the acceleration layer when displaying the touch track is higher than the display level of the graphics layer; the device interface is configured to connect to a second display, the second display is used to display the display screen of the graphics layer of the first display; the controller is configured to execute the following program steps:

[0007] In response to a first touch track input by a user, generating target bitmap data of the first touch track, wherein the first touch track is a touch track of a drawing event and includes at least one touch point;

[0008] Detecting a first touch point and a first reference touch point in the first touch track, where the first touch point is a touch point with the latest touch time in the first touch track;

[0009] Calculating a drawing area according to the first touch point and a first reference touch point;

[0010] The target bitmap data is refreshed to the acceleration layer and the graphics layer of the first display according to the drawing area, and the target bitmap data is transmitted to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

[0011] In a second aspect, some embodiments of the present application further provide a display device, including a first display, a device interface and a controller. The first display is configured to display a drawing interface, the drawing interface includes a display screen of a graphics layer, the first display is connected to a touch interaction module, the touch interaction module is used to detect a touch track input by a user, and when a touch track of an erase event input by a user is detected, the touch track of the erase event is also displayed through an acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released, and the display level of the acceleration layer when displaying the touch track is higher than the display level of the graphics layer; the device interface is configured to connect to a second display, the second display is used to display the display screen of the graphics layer of the first display; the controller is configured to execute the following program steps:

[0012] In response to a second touch track input by a user, detecting a bitmap background of the graphic layer, wherein the second touch track is a touch track of an erase event and includes at least one touch point;

[0013] Detecting a second touch point and a second reference touch point of the second touch track, wherein the second touch point is a touch point with the latest touch time in the second touch track, and the second reference touch point is a touch point with the shortest touch time from the second touch point in the second touch track;

[0014] Calculating an erasure area according to the second touch point and a second reference touch point;

[0015] The bitmap background is refreshed to the acceleration layer and the graphics layer of the first display according to the erased area, and the bitmap background is transmitted to the graphics layer of the second display according to the erased area, so that the second display displays the same display screen as the first display.

[0016] In a third aspect, some embodiments of the present application further provide a multi-screen synchronous touch display method, which is applied to the display device provided in the first aspect, and the method includes:

[0017] In response to a first touch track input by a user, generating target bitmap data of the first touch track, wherein the first touch track is a touch track of a drawing event and includes at least one touch point;

[0018] Detecting a first touch point and a first reference touch point of the first touch track, wherein the first touch point is a touch point with the latest touch time in the first touch track, and the first reference touch point is a touch point with the shortest touch time from the first touch point in the first touch track;

[0019] Calculating a drawing area according to the first touch point and the first reference touch point;

[0020] The target bitmap data is refreshed to the acceleration layer and the graphics layer of the first display according to the drawing area, and the target bitmap data is transmitted to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

[0021] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a multi-screen synchronous touch display method, and the method can generate target bitmap data of the first touch track in response to the first touch track input by the user. Wherein, the first touch track is the touch track of the drawing event, and the first touch track includes at least one touch point. Detect the first touch point with the latest touch time in the first touch track, and the first reference touch point with the shortest touch time from the first touch point, and calculate the drawing area according to the first touch point and the first reference touch point. Then refresh the target bitmap data to the acceleration layer and the graphics layer of the first display according to the drawing area, and transfer the target bitmap data to the graphics layer of the second display according to the drawing area. The method can synchronously refresh the bitmap data of the touch track to the acceleration layer and the graphics layer of the first display according to the touch point of the touch track, and transfer the bitmap data to the graphics layer of the second display, so that the first display and the second display display the same screen content, thereby improving the problem that the display screens between multiple displays cannot be synchronized. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0024] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0025] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0026] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0027] Figure 5 A schematic diagram of touch tracks provided in some embodiments of the present application;

[0028] Figure 6 A schematic diagram of a whiteboard application interface provided in some embodiments of the present application;

[0029] Figure 7 A structural diagram of a dual-screen display system provided for some embodiments of the present application;

[0030] Figure 8 A schematic diagram of a dual-screen display effect in a same-display mode provided in some embodiments of the present application;

[0031] Fig. 9 An example diagram of the asynchronous display effects of dual screens in the same display mode provided in some embodiments of the present application;

[0032] Fig.10 A schematic diagram of a flow chart of a multi-screen synchronous drawing and display method provided in some embodiments of the present application;

[0033] Fig.11 A schematic diagram of the effect of a first touch track in a first display provided in some embodiments of the present application;

[0034] Fig.12 A schematic diagram of the position of a drawing area in a touch track provided in some embodiments of the present application;

[0035] Fig.13 A schematic diagram of the effect of synchronously displaying a first touch track on multiple screens provided in some embodiments of the present application;

[0036] Fig.14 A schematic diagram of a process for drawing target bitmap data provided by some embodiments of the present application;

[0037] Fig.15 A schematic diagram of the effect of the eraser control trailing in the first display provided in some embodiments of the present application;

[0038] Fig.16 A schematic diagram of a flow chart of a multi-screen synchronous erasing display method provided in some embodiments of the present application;

[0039] Fig.17 A schematic diagram of the effect of a second touch track in a first display provided in some embodiments of the present application;

[0040] Fig.18 A schematic diagram of the effect of synchronously displaying a second touch track on multiple screens provided in some embodiments of the present application;

[0041] Fig.19 A schematic diagram of a process for updating a graphic layer bitmap background provided in some embodiments of the present application. DETAILED DESCRIPTION

[0042] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0043] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0044] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0045] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0046] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0047] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 may be a remote controller, a stylus pen, or the like.

[0048] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0049] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0050] In addition to providing the broadcast receiving television function, the display device 200 may also provide an intelligent network television function with computer support functions, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0051] Figure 2 Some embodiments of the present application provide Figure 12 is a block diagram of the hardware configuration of the display device 200.

[0052] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0053] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0054] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0055] In some embodiments, the display 260 is connected to a touch interaction module so that the display device 200 supports touch interaction operations. The touch interaction module can be a built-in module or an external module. For example, in order to implement touch interaction operations, for a display device 200 with a built-in touch interaction module, the display 260 can be a touch screen, which can receive touch signals input by fingers, contacts, styluses, etc., and execute touch interaction responses according to a pre-set interaction strategy.

[0056] For the display device 200 with an external touch interaction module, a terminal, device, or component with a touch function can be connected through the device interface of the display device 200. For example, the USB or USB-C interface of the display device 200 is connected to a touch panel, and the touch panel can receive the touch track input by the user in real time and transmit the touch track to the display device 200 through the USB channel. The display device 200 can perform an interactive response based on the received touch track in combination with the touch application, so that the display device 200 supports touch interaction operations.

[0057] In some embodiments, the communicator 220 is a component for communicating with an external device or a server 400 according to various communication protocol types.

[0058] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0059] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0060] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0061] In some embodiments, the user interface 280 is an interface that can be used to receive control input.

[0062] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0063] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0064] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0065] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0066] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0067] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0068] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0069] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0070] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0071] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0072] like Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0073] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0074] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0075] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0076] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0077] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0078] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] The display device 200 can also deploy other device interfaces, hardware, and drivers based on the kernel layer according to the specific display requirements of the display device 200. For example, in order to enrich the display types of the display device 200, an HDMI interface and a USB interface can be set on the display device 200, wherein the HDMI interface can connect devices with video and audio input / output functions, and implement the corresponding audio and video output / input functions through the HDMI driver. The USB interface can connect devices with signal and data transmission functions and interactive functions, and based on the USB driver, access signals, transmit data, and perform specific interactive actions.

[0080] For a display device 200 that supports touch interaction operations, the display device 200 has a built-in touch interaction module, and can receive the touch track input by the user through the touch interaction module. According to the interaction principle of the touch interaction module, the touch interaction module can detect relevant parameters in the touch action. For example, a capacitive touch layer is provided on the display 260 of the display device 200. When the user's finger touches any position of the touch layer, the capacitance of the touch layer at the touch position will change. At this time, the touch interaction module can record the position of the capacitance change, and record the time when the capacitance change occurs in combination with the timer in the controller to generate a touch track.

[0081] In some embodiments, the physical signal detected by the touch interaction module needs to be converted before it can be read by the controller 250 of the display device 200. For example, the analog signal of the capacitance change needs to be converted into a digital signal and then transmitted to the controller 250 of the display device 200. In order to adapt to the screen display process of the display device 200, the touch track detected by the touch interaction module also needs to be mapped to coordinates so that the position point corresponding to the touch track can have a corresponding relationship with the display 260 of the display device 200 or the pixel point in the user interface. For example, the coordinate position detection accuracy of the touch interaction module is 65536×65536, and the resolution of the display 260 of the display device 200 is 3840×2160. After the touch interaction module detects the touch track, the coordinate mapping of the touch point position in the width and height directions can be performed according to the ratio of 65536 / 3840 and 65536 / 2160, so that the display device 200 can perform related operation responses based on the mapped touch point position coordinates.

[0082] The touch track may include different operation parameters according to the user's input method. Figure 5As shown, the touch trajectory may include the touch time, touch position, touch event type, number of touch points, etc. of the touch operation. In a touch interaction process, the touch event types that can be detected by the touch interaction module include: down events, move events, and up events of touch actions. Correspondingly, on the entire touch trajectory, a starting point [down; x1, y1, t1], a trajectory point [move; x2, y2, t2], and an end point [up; x3, y3, t3] may be formed. Each touch trajectory records the corresponding touch event type, touch time t, and touch position (x, y), that is, the touch points of the touch trajectory may include a starting point, a trajectory point, and an end point.

[0083] It should be noted that, since the user's finger or touch device is in surface contact with the touch interaction module when performing touch interaction operations, that is, the touch interaction module can detect a contact area, and the contact area includes multiple contact points. Therefore, when the display device 200 performs interactive control, it can detect (or specify) a contact point in the contact area as a touch point. For example, when the contact area is an elliptical area formed when the user's finger contacts the touch interaction module, the display device 200 can traverse the contact point coordinates in the elliptical area, determine the coordinate extremes in two mutually perpendicular directions, to delineate the major axis and minor axis of the elliptical area, and use the contact point at the intersection of the major axis and the minor axis as the touch point.

[0084] In the process of executing the touch interaction response, the display device 200 can classify different touch times, touch positions, the number of touch points and the type of touch events, so as to detect the user's touch input gesture, thereby performing different interactive responses for different input gestures. For example, when the number of touch points in the touch trajectory detected by the touch interaction module is 1, the touch event type only includes a down event and an up event, and the time interval between the down event and the up event is less than the single-click event time threshold. In addition, if the touch position change distance of the down event and the up event is less than the jitter distance threshold, it can be determined that the current user's touch input gesture is a click gesture. Based on the click gesture, the display device 200 executes the operation of starting the application according to the icon in the area where the touch position corresponding to the click gesture is located.

[0085] Similarly, the display device 200 can also detect other touch interaction gestures input by the user based on the touch time, touch position, number of touch points and touch event type, such as double-click gesture, slide gesture, long press gesture, multi-finger click gesture, multi-finger slide gesture, etc. Different interaction gestures can derive different touch parameters. For example, a slide gesture can also generate parameters such as slide distance, slide speed, slide direction, slide track shape, etc. Different touch parameters can be used to perform different interactive controls.

[0086] In some embodiments, in order to enrich the touch interaction types supported by the display device 200, the display device 200 can also perform complex gesture detection based on the dynamic change rules of touch time, touch position, number of touch points and touch event type. For example, when the touch interaction module detects that the touch trajectory of the user input includes multiple touch points, and the touch position of each touch point approaches each other as the user inputs, it can be determined that the touch gesture currently input by the user is a grab gesture. Based on complex gesture detection, the display device 200 can perform more interactive responses. For example, when the detected touch gesture is a grab operation for an image view, the display device 200 can respond to the grab gesture and perform a control action to select and move the image display position.

[0087] In some embodiments, the touch gesture can also be combined with the touch position to form a touch interaction operation based on a specific position. For example, when the touch interaction module detects that the down event position during the touch operation is at the top of the display 260, the touch point position corresponding to the move event moves downward. When the up event is detected, the display device 200 can determine that the current touch gesture is to slide down from the top of the screen. Therefore, the display device 200 can respond to the touch gesture and control the display 260 to present a drop-down list.

[0088] Some touch interaction gestures can be used to perform interactive actions based on the operating system of the display device 200, which are called global gestures. For example, the operation of sliding down from the top of the screen to call out a drop-down list in the above example of the display device 200. Some touch interaction gestures can be used to perform interactive actions based on specific touch applications. For example, when the display device 200 runs a whiteboard application, the sliding touch operation input by the user can be used to draw a track line on the whiteboard.

[0089] In some embodiments, the display device 200 can generate bitmap data of the touch track in response to the touch track input by the user based on the touch application. When the end point of the touch track is detected, the bitmap data of the touch track is refreshed to the OSD layer of the display device 200, so that the display device 200 can present the display effect of the touch track.

[0090] The touch application can be a system application or a third-party application developed based on a specific display device type and user needs. The touch application can be installed in the display device 200 and started to run under user control. For example, the user can control the display device 200 to run the whiteboard application by clicking the "whiteboard" application icon in the application interface of the display device 200.

[0091] It should be noted that the whiteboard application can be an independent application, an associated application, or a functional module in a touch application. Different forms of whiteboard applications can have different startup or calling methods. For example, a user can start an independent whiteboard application through the application interface of the display device 200; a user can also start a whiteboard application by executing a "whiteboard" operation on a conference application; a user can also control the conference application to start the whiteboard functional module by clicking the "whiteboard" icon on the conference application.

[0092] Different forms of whiteboard applications may present different forms of whiteboard application interfaces after startup. For example, when a user starts a whiteboard application in an associated application form, the display device 200 may automatically enter a dual-screen (or split-screen) mode. In the dual-screen (or split-screen) mode, the display device 200 may display a user interface through two display areas, i.e., the main screen displays the original application (such as a conference application) interface, and the secondary screen displays the whiteboard application interface.

[0093] After the whiteboard application is running, the display device 200 can be switched to display a whiteboard interface. Figure 6 As shown, the whiteboard interface may include drawing tool controls for drawing operations, such as pen shape, graphics, line shape, color, erase, clear screen, selection, etc. The user can click any control to draw graphics according to specific drawing needs.

[0094] In some embodiments, the display device 200 classifies the touch track input by the user based on the whiteboard application according to the selected state of the drawing tool control. For example, when the drawing tool control such as pen shape, graphic shape, line shape, etc. is in the selected state, the touch track input by the user is marked as a drawing event; when the erasing drawing tool is in the selected state, the touch track input by the user is marked as an erasing event. The display device 200 then responds to the touch track of the drawing event or the erasing event and executes different programs according to the preset interaction strategy.

[0095] In some embodiments, the display device 200 can form a dual-screen system by connecting an external display to the display 260 via the HMDI output to achieve a dual-screen display mode. Figure 7As shown, the dual-screen system may include an image generation (ImageProducer) module, an image rendering (surface flinger) module, a synthesis (Composer) module, a video process (VideoProcessing Unit hardware, VPU hardware) module, a main screen (panel screen) and a secondary screen (HDMI screen).

[0096] During the display process, the media player, camera peripherals, and graphics interface (openGLES) of the image generation module can generate image content and pass the image content to the image rendering module. The image rendering module includes a primary display unit and an extended display unit. The primary display unit is used to control the display content of the main screen, that is, the primary display unit renders the image content through the user interaction layer (UI layer) or the video layer according to the image content.

[0097] In order to obtain a better display effect, the image content of the UI layer can be rendered by the GPU component, so as to form a picture frame for display in the synthesis module. Among them, the picture frames can be stored in the synthesis module through the frame buffer and the video buffer respectively, and the picture content is formed in the corresponding layers through the video process module. That is, the image frames generated by the video playback process form the picture content through the video plane, and the image frames of other image pictures (such as touch tracks, graphics, etc.) form the picture content through the primary OSD plane. Finally, the display device 200 forms the final picture through the blending unit of the video process module and displays it on the main screen.

[0098] The extended display unit is used to control the display content of the secondary screen, that is, after the image generation module passes the image content to the image rendering module, the extended display unit also renders the image content through the UI layer or the video layer. According to the display process of the main screen, the image frame is formed by the image rendering module, the synthesis module and the video process module respectively, and displayed on the secondary screen.

[0099] In some embodiments, in order to obtain a better picture display effect, an additional extended display layer (external OSD plane) can be set in the video process module of the display device 200 on the basis of the main OSD layer and the video layer. The extended display layer can be configured with different layer characteristics according to specific display effect requirements. For example, the bitmap refresh mode of the extended display layer can be modified so that the extended display layer is refreshed in real time according to the region, thereby improving the display speed of the graphics. At this time, the extended display layer can be used as an acceleration layer in scenes with high dynamic display effects.

[0100] Taking the display device 200 running a whiteboard drawing application as an example, in order to improve the hand tracking of the line drawing process, after detecting the user's touch operation, the bitmap data of the touch track can be refreshed in real time in the extended display layer (acceleration layer) according to the bitmap area composed of two adjacent touch points of the touch track. The display level of the acceleration layer can be higher than other layers, and the display speed is higher than other layers, thereby improving the real-time display effect of the whiteboard application drawing process.

[0101] In some embodiments, the extended display layer can also work together with other graphics to improve display quality. For example, the acceleration layer can respond to the touch track input by the user at the same time as the main OSD layer, and draw graphics separately. However, since the acceleration layer is displayed above the main OSD layer, the graphics drawn by the acceleration layer can block the graphics drawn by the OSD layer to maintain real-time response speed. After the user inputs the touch interaction action, that is, when the up event is monitored, the content displayed by the acceleration layer can be released directly, and the OSD layer directly forms the drawing graphics, so there is no need for the acceleration layer to refresh the drawing graphics to the OSD layer, reducing the graphics mixing time. It not only improves the real-time display effect of the graphics drawing process, but also improves the final presentation quality of the graphics drawing.

[0102] Based on the above dual-screen display system, the user can control the display device 200 to execute different display strategies as needed. For example, the main screen is connected to the secondary screen through HDMIOUT to realize dual-screen display, and the secondary screen integrates a touch component to realize touch-control of the secondary screen content. In addition, the main screen and the secondary screen are also connected via USB, and the USB connection channel can be used to transmit the touch track, that is, the secondary screen touch is connected to the USB of the main screen. After the connection is completed, the display device 200 can control the display of the main screen and the secondary screen. Among them, the main screen and the secondary screen can support the same display, that is, the display 260 (main screen) of the display device 200 and the external display display the same content. The main screen and the secondary screen can also support different displays, that is, the main screen and the secondary screen display different contents. Different displays are also called extended display. The main scenarios include: the physical signal source screen is displayed on the secondary screen; the player screen is displayed on the secondary screen; the wireless screen transmission screen displays two screen transmission screens; the conference application displays the shared content on the secondary screen, etc.

[0103] The following describes the display process based on the above-mentioned dual-screen system through specific examples. It should be understood that the following display process is only an example and does not limit the specific display method of the dual-screen system. Other display methods and interface contents that can be associated with by technical personnel in this field based on the following examples all belong to the protection scope of this application.

[0104] like Figure 8 As shown, according to the above-mentioned dual-screen system connection method, after the main screen is connected to the secondary screen via an HDMI cable, the dual-screen mode can be started. In the dual-screen mode, the display device 200 can set the display mode of the secondary screen. The display mode defaults to the same display mode, that is, the secondary screen displays the same display screen as the main screen.

[0105] Based on the above embodiment, the extended display layer of the main screen can refresh the bitmap data of the touch track in real time during the drawing process of the touch track, so that the main screen can display the real-time drawing process of the touch track in real time through the extended display layer. Figure 7 In the dual-screen display system shown in FIG. 1 , since the secondary screen does not have an extended display layer, the secondary screen cannot display the drawing process of the touch track, and can only display the drawing result of the touch track through the OSD layer. The display device 200 refreshes the bitmap data of the touch track to the OSD layer only when the end point of the touch track is detected. In this way, in the same display mode, Fig. 9 As shown, during the process of the user inputting the touch trajectory, before the display device 200 detects the end point of the touch trajectory, that is, before the user raises his hand, the main screen can display the touch trajectory in real time through the acceleration layer, but the secondary screen cannot display the touch trajectory, resulting in the main screen and the secondary screen being unable to synchronize their display images, reducing the user experience.

[0106] For ease of distinction, in the embodiment of the present application, the main screen of the display device 200 is referred to as a first display, and the sub-screen is referred to as a second display.

[0107] Based on the above application scenarios, in order to improve the problem that the display screens of the main screen and the secondary screen in the display device 200 cannot be synchronized, some embodiments of the present application provide a display device 200, including a first display 261, a device interface 240 and a controller 250. Among them, the first display 261 is configured to display a drawing interface, and the drawing interface includes the display content of the graphics layer, and the graphics layer is the OSD layer described in the above embodiment. The first display 261 is connected to a touch interaction module, and the touch interaction module is used to detect the touch track of the user input. In addition, when the touch track of the user input drawing event is detected, the first display 261 also displays the touch track of the drawing event through the acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released, and the acceleration layer is the extended display layer described in the above embodiment; when the acceleration layer displays the touch track, the display level is higher than the display level of the graphics layer, and the display speed of the acceleration layer is higher than the display speed of the graphics layer. The device interface 240 is configured to connect to the second display 262, and the second display 262 is used to display the display screen of the graphics layer of the first display 261.

[0108] like Fig.10 As shown, the controller 250 is configured to perform the following program steps:

[0109] S110: In response to a first touch track input by a user, generating target bitmap data of the first touch track.

[0110] Among them, the first touch trajectory is the touch trajectory of the drawing event, and the first touch trajectory includes at least one touch point. The user can input the first touch trajectory of the drawing event to the display device 200 through a touch application such as a whiteboard. The first touch trajectory is a touch data set formed by at least one touch point generated by the user performing a touch operation. The touch trajectory can be a dynamic data set. For example, after the user selects the pen-shaped control of the whiteboard application, the display device 200 starts recording the touch trajectory after detecting a down event located in the drawing or writing area. As the user slides, the touch interaction module will detect multiple touch points of move events, and the touch trajectory is a touch point set composed of the touch point of the down event and all the touch points of the move event from the down event to the current time. The display device 200 will collect the touch points of the move event at intervals of the time period according to a preset time period.

[0111] After detecting the first touch track of the drawing event, the display device 200 generates target bitmap data of the first touch track in response to the first touch track. The target bitmap data is bitmap data generated based on each touch point in the first touch track, and the bitmap data is a data format that can be displayed in a bitmap. For example, the touch point in the touch track can be converted into a pixel, and each pixel is provided with specific touch color information.

[0112] In some embodiments, the touch color information includes color values, and the color values ​​include color values ​​of four channels: R (red), G (green), B (blue), and A (transparency). The display device 200 can set corresponding color values ​​for pixels according to the touch color information selected by the user. For example, after the user clicks the red option on the color palette and enters the first touch track in the writing or drawing area, the display device 200 can set touch color information including a color value of red for each pixel converted by the first touch track.

[0113] S120: Detect a first touch point and a first reference touch point of a first touch track.

[0114] The first touch point is the touch point with the latest touch time in the first touch track, and the first reference touch point is the touch point with the shortest touch time from the first touch point in the first touch track. When generating the target bitmap data of the first touch track, the display device 200 also detects the first touch point and the first reference touch point in the first touch track in real time, so as to determine the area drawn in real time by the first touch point and the first reference touch point. For example, Fig.11 The touch track of the drawing event shown in FIG. 1 is the starting point of the touch track, and the touch point bf is the track point of the touch track. The user draws the touch track on the first display 261 of the display device 200 by touching with a finger. Fig.11 When the touch point is at the position shown, the first touch point detected by the display device 200 is point f.

[0115] S130: Calculating a drawing area according to the first touch point and the first reference touch point.

[0116] After the display device 200 detects the first touch point of the first touch track, it calculates the drawing area that needs to be refreshed according to the first touch point and the first reference touch point. The drawing area is used to determine the area position where the target bitmap data needs to be refreshed in real time. Since the display device 200 collects the touch points of the first touch track at intervals, the display device 200 can refresh the target bitmap data according to the two touch points with adjacent touch times in the first touch track to update the display screen of the touch track in real time. In order to achieve the effect of real-time update, the two adjacent touch points can use the first touch point with the latest touch time in the first touch track, and the touch point with the shortest touch time distance from the first touch point in the first touch track.

[0117] Therefore, in some embodiments, the display device 200 creates a rectangular object when calculating the drawing area. The rectangular object is used to store parameters that appear in pairs. The rectangular object (Rect) includes rectangular parameters, and the rectangular parameters include first vertex coordinates and second vertex coordinates. The first vertex coordinates and the second vertex coordinates are the coordinates of the diagonal vertices of the rectangle, such as the upper left corner coordinates and the lower right corner coordinates of the rectangle. The position coordinates of the first touch point and the first reference touch point are read, and the position coordinates of the first touch point are set as the first vertex coordinates, and the position coordinates of the first reference touch point are set as the second vertex coordinates. Then, the rectangular area is delineated according to the rectangular object to obtain the drawing area.

[0118] For example, Fig.11 As shown, the first touch point is f, the first reference touch point is e, and the position coordinates of the first touch point f are (x 1 ,y 1 ), the position coordinates of the first reference touch point e are (x 2 ,y 2 ). The user inputs to the first display 261 of the display device 200 Fig.11 In the touch track shown in the figure, the display device 200 creates a Rect object and initializes the rectangular parameters of the Rect object to (left, top, right, bottom); where left and top are the first vertex coordinates, which are the coordinates of the upper left corner of the rectangular object; right and bottom are the second vertex coordinates, which are the coordinates of the lower right corner of the rectangular object. When the user moves from the touch point e to the touch point f, the display device 200 refreshes the target bitmap data and sets (x 1 ,y 1 ) is set as the first vertex coordinate and (x 2 ,y 2 ) is set to the coordinates of the second vertex, that is, the rectangular parameters of the Rect object are set to (x 1 ,y 1， x 2 ,y 2 ). At this time, the rectangular area formed by the Rect objects, that is, the drawing area, is as follows Fig.12 As shown, the display device 200 needs to refresh the target bitmap data of the drawing area.

[0119] In some embodiments, the rectangle parameters of the rectangle object may also include other attributes, which may be used to perform operations in any rectangular range. For example, the rectangle parameters may also include center coordinates, rotation angles, and transparency.

[0120] S140: Refreshing the target bitmap data to the acceleration layer and the graphics layer of the first display according to the drawing area, and transmitting the target bitmap data to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

[0121] After calculating the above-mentioned drawing area, the display device 200 refreshes the target bitmap data to the acceleration layer and the graphics layer of the first display 261 according to the drawing area, and transmits the target bitmap data to the graphics layer of the second display 262 according to the drawing area. In this way, by synchronously refreshing the acceleration layer and the graphics layer, the acceleration layer and the graphics layer of the first display 261 and the graphics layer of the second display 262 can simultaneously receive the target bitmap data of the same drawing area, and display the target bitmap data of the drawing area in real time, thereby ensuring that the first display 261 and the second display 262 can display the same screen content, such as Fig.13 shown.

[0122] In the embodiment of the present application, the graphic layer of the second display 262 and the graphic layer of the first display 261 use the same interaction logic, that is, in the same display mode, the bitmap data received by the second display 262 and the bitmap data received by the first display 261 are bitmap data of the same source. Therefore, in the embodiment of the present application, the graphic layers of the first display 261 and the second display 262 are not distinguished.

[0123] When the display device 200 refreshes the target bitmap data according to the drawing area, the SurfaceHol der interface of the display device 200 can be called to lock the canvas (Canvas) used to draw the target bitmap data, so as to prevent other threads from affecting the drawing of the target bitmap data. Therefore, in some embodiments, when the display device 200 refreshes the target bitmap data to the graphics layer, the canvas of the surface view is locked based on the rectangular object. Among them, the surface view (SurfaceView) is used to run the canvas, and the canvas is used to draw the target bitmap data. The surface view is an inherited class of the view system (View System). The drawing operation of the surface view is performed in an independent rendering thread, does not occupy the main thread resources, and can improve the drawing efficiency of the target bitmap data. After locking the canvas, the display device 200 draws the target bitmap data of the rectangular area through the canvas, and after the canvas finishes drawing the target bitmap data of the rectangular area, unlocks the canvas and submits the canvas to the surface view. The canvas is then submitted to the window manager (Window Manager) through the surface view, so that the canvas is rendered to the graphics layer of the first display 261 through the window manager, and the canvas is transmitted to the graphics layer of the second display 262.

[0124] For example, the first touch point is f, the reference touch point is e, the position coordinates of the first touch point f are (x1, y1), and the position coordinates of the reference touch point e are (x2, y2). When refreshing the target bitmap data, the display device 200 locks the canvas through the SurfaceHolder interface and defines the rectangular parameters of the rectangular object Rect in the canvas. The canvas of the display device 200 draws the target bitmap data in the area of ​​the Rect object, and after the drawing is completed, it unlocks the canvas through the SurfaceHolder interface and submits the canvas to the surface view. The surface view then passes the canvas to the window manager, and the window manager writes the target bitmap data drawn on the canvas into the graphics layer, so that the first display 261 and the second display 262 can display the same display screen.

[0125] In some application scenarios, the bitmap data used by the acceleration layer and the graphics layer of the display device 200 when displaying the touch track can be the same bitmap data or different bitmap data, that is, the bitmap data refreshed to the acceleration layer and the bitmap data refreshed to the graphics layer can be from the same source or from different sources. When the acceleration layer and the display layer share the same bitmap data, the display device 200 can directly refresh the acceleration layer and the graphics layer through the bitmap data used by the acceleration layer and the display layer, so that the display screens of the acceleration layer and the display layer remain consistent.

[0126] Therefore, in some embodiments, if the acceleration layer and the graphics layer use the same bitmap data, the display device 200 maps the target bitmap data to the reference bitmap data when refreshing the target bitmap data. The reference bitmap data is the bitmap data used by the acceleration layer and the graphics layer. The reference bitmap data of the drawing area is detected to obtain the first refresh bitmap data. The first refresh bitmap data is then mapped to the graphics layer and acceleration layer of the first display 261 and the graphics layer of the second display 262. The target bitmap data is preset with a fixed mapping relationship with the graphics layer and the acceleration layer. After determining the first refresh data of the reference bitmap data in the drawing area, the display device 200 maps the first refresh data to the graphics layer of the first display 261, and transmits the first refresh data to the graphics layer of the second display 262, so that the second display 262 can display the same display screen as the first display 260.

[0127] In order to speed up the response speed of the acceleration layer, in some embodiments, the acceleration layer and the graphics layer use different bitmap data, namely, upper bitmap data and lower bitmap data. When the user inputs a touch track, the display device 200 writes the bitmap data of the touch track into the upper bitmap data and the lower bitmap data. Among them, the upper bitmap data is the bitmap data used by the acceleration layer, and the lower bitmap data is the bitmap data used by the graphics layer. In the process of refreshing the acceleration layer and the graphics layer, the data volume of the upper bitmap data may be smaller than the data volume of the lower bitmap data.

[0128] For example, when a user inputs a first touch trajectory to the display device 200, the upper-layer bitmap data only stores bitmap data of one touch trajectory, that is, after the display device 200 refreshes the target bitmap data to the acceleration layer, the upper-layer bitmap data releases the target bitmap data, and when the user draws a new touch trajectory, the new bitmap data of the touch trajectory is generated by the upper-layer bitmap data; the lower-layer bitmap data can store the bitmap data of all touch trajectories, that is, after the display device 200 refreshes the target bitmap data to the graphics layer, the lower-layer bitmap data still retains the target bitmap data, and when the user draws a new touch trajectory, the bitmap data of the new touch trajectory continues to be drawn on the upper layer of the target bitmap data.

[0129] For the above scenario, when the bitmap data used by the acceleration layer and the graphics layer are different, in order to make the display screens of the graphics layer and the acceleration layer consistent, the bitmap data used by the acceleration layer and the bitmap data used by the graphics layer need to be superimposed. Therefore, in some embodiments, when the acceleration layer and the graphics layer use different bitmap data, the display device 200 maps the target bitmap data to the upper layer bitmap data and obtains the lower layer bitmap data when refreshing the target bitmap data. Among them, the upper layer bitmap data is the bitmap data used by the acceleration layer, and the lower layer bitmap data is the bitmap data used by the graphics layer. The upper layer bitmap data is superimposed on the upper layer of the lower layer bitmap data to generate superimposed bitmap data. Then the superimposed bitmap data of the drawing area is detected to obtain the second refresh bitmap data. The second refresh bitmap data is mapped to the graphics layer of the first display 261, and the second refresh data is transmitted to the graphics layer of the second display 262. By superimposing the bitmap data of the acceleration layer and the graphics layer, the second display 262 can display the same display screen as the first display 260.

[0130] For example, the display device 200 locks the canvas through the SurfaceHolder interface and defines the rectangular parameters of the rectangular object Rect in the canvas so that the canvas of the display device 200 draws the bitmap data in the area of ​​the Rect object. At this time, the bitmap data drawn by the canvas may include three layers, such as Fig.14 As shown, the bitmap data of the bottom layer is the background, the bitmap data of the middle layer is the bitmap data of the graphic layer, and the bitmap data of the highest layer is the bitmap data of the acceleration layer.

[0131] In order to facilitate the user to perform touch operations in the display device 200, in some embodiments, the display device 200 also responds to the user's selection operation on a specific control and displays the bitmap data of the prompt line in real time through the acceleration layer. For example, after the user selects a control such as circle selection in the drawing tool control, the display device 200 generates the bitmap data of the prompt line corresponding to the circle selection according to the user's touch trajectory, and directly refreshes it to the acceleration layer.

[0132] The above-mentioned prompt line is used to assist the user in performing touch operations, and is not the line that the user wants to draw. Therefore, after completing operations such as selection, circling, and rubbing, the acceleration layer will release the bitmap data of the prompt line to ensure that the prompt line does not affect the display screen of the drawing interface. However, when the display device 200 refreshes the target bitmap data of the first touch track to the graphics layer, the acceleration layer and the graphics layer will use the bitmap data superimposed. In this way, after the acceleration layer releases the bitmap data of the prompt line, the bitmap data of the prompt line will still remain in the bitmap data of the graphics layer, causing it to interfere with the display screen of the drawing interface.

[0133] Therefore, in order to improve the interference problem of the above-mentioned prompt line, in some embodiments, the display device 200 also monitors the touch point of the first touch track. Among them, the touch point includes the above-mentioned starting point, track point and end point. When the end point of the first touch track is detected, the bitmap data of the canvas is cleared, and the first target area is calculated according to the starting point and the end point of the first touch track. Based on the first target area, the target bitmap data is drawn through the canvas, that is, the display device 200 can redraw the bitmap data of the graphics layer after completing the drawing operation.

[0134] When redrawing the bitmap data of the graphics layer, in order to achieve the effects of selection, circling and erasing, in some embodiments, the display device 200 detects the selection result of the first editing instruction in response to the first editing track. Among them, the first editing track is a touch track for selection, that is, the user can select the drawn touch track in the drawing interface through the first editing track. When the end point of the first editing track, that is, the touch point of the up event, is detected, the display device 200 can obtain the corresponding selection result based on each touch point of the first editing track. For example, the touch tracks that intersect with the first editing track and are within the closed range formed by the first editing track are all touch tracks selected by the first editing track. After detecting the selection result of the first editing track, the display device 200 redraws the bitmap data of the graphics layer, and performs stroke mark drawing on the bitmap data corresponding to the selection result to present the effect of the touch track selection.

[0135] Similarly, in some embodiments, for the second editing instruction for performing circle erasing. The display device 200 responds to the second editing instruction and detects the circle erasing area of ​​the second editing instruction. When the end point of the second editing track, that is, the touch point of the up event, is detected, the display device 200 can obtain the corresponding circle erasing area based on each touch point of the second editing track. After the display device 200 detects the circle erasing area of ​​the second editing track, it redraws the bitmap data of the remaining area of ​​the graphics layer except the circle erasing area to present the effect of performing circle erasing on the touch track. For example, the display device 200 locks the canvas through the SurfaceHolder interface, and defines the rectangular parameters of the rectangular object Rect in the canvas, so that the canvas of the display device 200 draws the bitmap data in the area of ​​the Rect object. After the drawing is completed, the display device 200 unlocks the canvas through the SurfaceHolder interface so that the canvas can be called normally by the drawing thread of the graphics layer, and the graphics layer can redraw the bitmap data based on the canvas.

[0136] In some embodiments, the display device 200 also enters a clear mode in response to the second touch track of the erase event. In the clear mode, the bitmap data at the position of the second touch track is set to a transparent bitmap to achieve the erasing effect of the second touch track. For example, after the user performs a selection operation on the erase control of the whiteboard application, the display device 200 can enter the clear mode based on the Porter-Duff algorithm, and in the clear mode, the bitmap data at the position of the second touch track is set to a transparent bitmap, that is, the color value of the bitmap data is set to 0.

[0137] In some embodiments, in order to prompt the user to input the second touch track of the erase event, the display device 200 also generates bitmap data of the eraser control, and refreshes the data of the eraser control to the acceleration layer and the graphics layer in real time according to the second touch point of the second touch track. Wherein, the second touch point is the touch point with the latest touch time in the second touch track.

[0138] However, since the display speed of the acceleration layer is higher than that of the graphics layer, when the display device 200 refreshes the touch track of the erasing event to the acceleration layer and the graphics layer at the same time, the acceleration layer of the first display 261 will more quickly set the bitmap data of the touch track position to a transparent bitmap. At this time, since the display speed of the graphics layer is slower than that of the acceleration layer, the moving position of the blackboard eraser control in the graphics layer will lag behind the moving position of the blackboard eraser control in the acceleration layer, that is, the positions of the blackboard eraser controls in the graphics layer and the acceleration layer are different. In this way, after the acceleration layer sets the bitmap data to the transparent bitmap, the user will observe the blackboard eraser control of the graphics layer through the transparent bitmap, and at the same time observe the blackboard eraser control of the acceleration layer through the acceleration layer, so that the first display 261 appears as follows. Fig.15 The eraser trailing problem shown.

[0139] Based on the above embodiments, in order to improve the tailing problem, some embodiments of the present application also provide a display device 200, including a first display 261, a device interface 240 and a controller 250. Among them, the first display 261 is configured to display a drawing interface, and the drawing interface includes the display content of the graphics layer, and the graphics layer is the OSD layer described in the above embodiments. The first display 261 is connected to a touch interaction module, and the touch interaction module is used to detect the touch track of the user input. In addition, when the touch track of the user input erase event is detected, the first display 261 also displays the touch track through the acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released. The acceleration layer is the extended display layer described in the above embodiments; when displaying the touch track, the display level of the acceleration layer is higher than the display level of the graphics layer, and the display speed of the acceleration layer is higher than the display speed of the graphics layer. The device interface 240 is configured to connect to the second display 262, and the second display 262 is used to display the display screen of the graphics layer of the first display 261.

[0140] like Fig.16 As shown, the controller 250 is configured to perform the following program steps:

[0141] S210: In response to a second touch track input by the user, detecting a bitmap background of a target graphic layer.

[0142] Among them, the second touch track is the touch track of the erase event, and the second touch track includes at least one touch point. The user can input the second touch track of the erase event to the display device 200 through a touch application such as a whiteboard. The second touch track is a touch data set formed by at least one touch point generated by the user performing a touch operation. The touch track can be a dynamic data set. For example, after the user selects the erase control of the whiteboard application, the display device 200 starts recording the touch track after detecting a down event located in the drawing or writing area. As the user slides, the touch interaction module will detect multiple touch points of move events, and the touch track is a touch point set consisting of the touch point of the down event and all the touch points of the move event from the down event to the current time. The display device 200 will collect the touch points of the move event at intervals of the time period according to a preset time period.

[0143] Since the acceleration layer of the first display 261 does not exist in the second display 262, in the same display mode, the second display 262 cannot present the real-time display effect of the acceleration layer on the second touch track. Therefore, after detecting the second touch track with the erase event, the display device 200 will detect the bitmap background of the target graphic layer in response to the second touch track. Among them, the target graphic layer includes the graphic layer of the first display 261 and the graphic layer of the second display 262. The target graphic layer is used to display the application window of the touch application. When detecting the bitmap background of the target graphic layer, the display device 200 can read the background attributes pre-stored in the touch application to obtain the bitmap background of the target graphic layer. Since the data used by the first display 261 and the second display 262 are the same source data, detecting the bitmap background of the target bitmap data is to detect the bitmap background of the graphic layer of the first display 261 or the graphic layer of the second display 262.

[0144] S220: Detect a second touch point and a second reference touch point of a second touch track.

[0145] The second touch point is the touch point with the latest touch time in the second touch track. When generating the target bitmap data of the first touch track, the display device 200 also detects the first touch point of the touch time in the first touch track in real time, so as to determine the area drawn in real time by the first touch point. Fig.17 The touch track of the erase event shown in FIG. 1 is a starting point of the touch track, and the touch point hl is a track point of the touch track. The user draws the touch track on the first display 261 of the display device 200 by touching with a finger. Fig.17 When the position shown is , the first touch point detected by the display device 200 is point 1. Fig.17 The dotted line part in the is used to represent the touch track of the erase event and will not be displayed in the drawing interface.

[0146] S230: Calculate an erasure area according to the second touch point and the second reference touch point.

[0147] After the display device 200 detects the second touch point of the second touch track, it calculates the erasure that needs to be refreshed according to the second touch point, and the erasure area is used to determine the location of the area that needs to be erased in the bitmap data of the graphic layer. Since the display device 200 collects the touch points of the second touch track at intervals, the display device 200 can calculate the erasure area according to the two touch points with adjacent touch times in the second touch track to update the erasure effect of the touch track in real time. In order to meet the real-time updated erasure effect, the two adjacent touch points can use the second touch point with the latest touch time in the second touch track, and the touch point with the shortest touch time distance from the second touch point in the second touch track.

[0148] S240: Refreshing the bitmap background to the acceleration layer and the graphics layer of the first display according to the erased area, and transmitting the bitmap background to the graphics layer of the second display according to the erased area, so that the second display displays the same display screen as the first display.

[0149] After calculating the erased area, the display device 200 refreshes the bitmap to the acceleration layer and the graphics layer of the first display 261 according to the erased area, and transmits the bitmap background to the graphics layer of the second display 262 according to the erased area. In this way, the bitmap background can cover the bitmap data of the lower layer according to the second touch track to present an erased effect.

[0150] The display device 200 synchronously refreshes the bitmap background in the acceleration layer and the graphics layer, so that the acceleration layer and the graphics layer of the first display 261, and the graphics layer of the second display 262 can simultaneously receive the bitmap background of the same erasing area, and display the bitmap background of the erasing area in real time to cover the bitmap data of the lower layer to achieve the erasing effect. At the same time, refreshing the bitmap background to the erasing area of ​​the acceleration layer can also make the acceleration layer block the eraser control of the graphics layer, thereby improving the trailing problem of the eraser control. Fig.18 As shown, the display device 200 provided in the embodiment of the present application ensures that the first display 261 and the second display 262 can display the same screen content and effectively improves the tailing problem of the eraser control.

[0151] It is understandable that the calculation method of the erased area in the embodiment of the present application can be consistent with the calculation method of the drawn area, which will not be described in detail here.

[0152] However, when using the display device 200, the user can also start the annotation mode based on the whiteboard application. In the annotation mode, the whiteboard application interface used to render the touch track has a transparent background, and the user can directly see the display screen of the display device 200 through the transparent background. However, the transparent background cannot cover the touch track and cannot achieve the effect of erasing.

[0153] Therefore, in some embodiments, Fig.19 As shown, the display device 200 also takes a screenshot of the drawing interface in response to the start instruction of the annotation mode to generate a screenshot of the annotation mode background. The background screenshot is then set as the bitmap background of the target graphics layer, thereby improving the problem that the graphics layer bitmap background blocks the annotation view. In other words, the display device 200 takes a screenshot of the drawing interface every time the annotation mode is started, and updates the bitmap background of the graphics layer through the image generated by the screenshot.

[0154] In some embodiments, since the display device 200 also has other menu controls, such as a clock control, a drawing tool control, etc., the above controls are views used to prompt the user to perform operations, and are not touch tracks drawn by the user. Therefore, the display device 200 also obtains the bitmap data of the menu control, and draws the bitmap data of the menu control to the acceleration layer, so that the touch track of the erase event does not interfere with the menu control.

[0155] Similarly, in order to reduce interference from factors such as prompt lines and eraser controls, in some embodiments, the display device 200 also monitors the touch points of the second touch track, and the touch points include a starting point, a track point, and an end point. When the end point of the second touch track is detected, the second target area is calculated according to the end point and the starting point of the second touch track. Then, according to the second target area, the bitmap data of the target graphics layer is set to a transparent bitmap, and the bitmap data of the acceleration layer is released, that is, when the display device 200 detects the end point of the second touch track, the bitmap data of the acceleration layer is released, and the bitmap data of the second touch track position in the graphics layer is set to a transparent bitmap.

[0156] Based on the above display device 200, some embodiments of the present application further provide a multi-screen synchronous touch display method, which is applied to the above display device 200, such as Fig.10 As shown, the method comprises the following procedural steps:

[0157] S110: generating target bitmap data of a first touch track in response to a first touch track input by a user, wherein the first touch track is a touch track of a drawing event and includes at least one touch point;

[0158] S120: Detecting a first touch point and a first reference touch point of the first touch trajectory, wherein the first touch point is a touch point with the latest touch time in the first touch trajectory, and the first reference touch point is a touch point with the shortest touch time from the first touch point in the first touch trajectory;

[0159] S130: calculating a drawing area according to the first touch point and the first reference touch point;

[0160] S140: Refreshing the target bitmap data to the acceleration layer and the graphics layer of the first display according to the drawing area, and transmitting the target bitmap data to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

[0161] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a multi-screen synchronous touch display method, and the method can generate target bitmap data of the first touch track in response to the first touch track input by the user. Wherein, the first touch track is the touch track of the drawing event, and the first touch track includes at least one touch point. Detect the first touch point with the latest touch time in the first touch track, and the first reference touch point with the shortest touch time from the first touch point, and calculate the drawing area according to the first touch point and the first reference touch point. Then refresh the target bitmap data to the acceleration layer and the graphics layer of the first display according to the drawing area, and transfer the target bitmap data to the graphics layer of the second display according to the drawing area. The method can synchronously refresh the bitmap data of the touch track to the acceleration layer and the graphics layer of the first display according to the touch point of the touch track, and transfer the bitmap data to the graphics layer of the second display, so that the first display and the second display display the same screen content, thereby improving the problem that the display screens between multiple displays cannot be synchronized.

[0162] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0163] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A display device, characterized in that: include: A first display is configured to display a drawing interface, wherein the drawing interface includes a display screen of a graphics layer; The first display is connected to a touch interaction module, and the touch interaction module is used to detect a touch track of a user input; when the touch track of a drawing event input by the user is detected, the first display also displays the touch track of the drawing event through an acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released, and the display level of the acceleration layer when displaying the touch track is higher than the display level of the graphics layer; A device interface, configured to connect to a second display, the second display being used to display a display screen of the graphic layer of the first display; The controller is configured as: In response to a first touch track input by a user, generating target bitmap data of the first touch track, wherein the first touch track is a touch track of a drawing event and includes at least one touch point; Detecting a first touch point and a first reference touch point of the first touch track, wherein the first touch point is a touch point with the latest touch time in the first touch track, and the first reference touch point is a touch point with the shortest touch time from the first touch point in the first touch track; Calculating a drawing area according to the first touch point and the first reference touch point; The target bitmap data is refreshed to the acceleration layer and the graphics layer of the first display according to the drawing area, and the target bitmap data is transmitted to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

2. The display device according to claim 1, characterized in that The controller calculates a drawing area according to the first touch point and the first reference touch point, and is configured to: Create a rectangle object, the rectangle object includes rectangle parameters, the rectangle parameters include first vertex coordinates and second vertex coordinates, the first vertex coordinates and the second vertex coordinates are the coordinates of diagonal vertices of the rectangle; Reading the position coordinates of the first touch point and the first reference touch point; Setting the position coordinates of the first touch point as the first vertex coordinates, and setting the position coordinates of the first reference touch point as the second vertex coordinates; A rectangular area is defined according to the rectangular object to obtain the drawing area.

3. The display device according to claim 2, characterized in that The controller is also configured to: Locking a canvas of a surface view based on the rectangular object, the surface view is used to run the canvas, and the canvas is used to draw the target bitmap data; Drawing the target bitmap data of the rectangular area through the canvas; After the canvas finishes drawing the target bitmap data of the rectangular area, unlocking the canvas and submitting the canvas to the surface view; The canvas is submitted to a window manager through the surface view so that the canvas is rendered to a graphics layer of the first display through the window manager and the canvas is transferred to a graphics layer of the second display.

4. The display device according to claim 3, characterized in that The controller is also configured to: Monitoring the touch points of the first touch track, wherein the touch points include a starting point, a track point, and an end point; When the end point of the first touch track is detected, clearing the bitmap data of the canvas, and calculating the first target area according to the start point and the end point of the first touch track; Based on the first target area, the target bitmap data is drawn through the canvas.

5. The display device according to claim 1, characterized in that If the acceleration layer and the graphics layer of the first display use the same bitmap data, the controller is further configured to: Mapping the target bitmap data to reference bitmap data, wherein the reference bitmap data is the bitmap data used by the acceleration layer and the graphics layer; Detecting the reference bitmap data of the drawing area to obtain first refresh bitmap data; The first refresh bitmap data is mapped to a graphics layer of the first display, and the first refresh bitmap data is transmitted to a graphics layer of the second display.

6. The display device according to claim 1, characterized in that If the acceleration layer and the graphics layer of the first display use different bitmap data, the controller is further configured to: Mapping the target bitmap data to upper-layer bitmap data, and acquiring lower-layer bitmap data, wherein the upper-layer bitmap data is the bitmap data used by the acceleration layer, and the lower-layer bitmap data is the bitmap data used by the graphics layer; superimposing the upper layer bitmap data onto the upper layer of the lower layer bitmap data to generate superimposed bitmap data; detecting the superimposed bitmap data of the drawing area to obtain second refresh bitmap data; The second refresh bitmap data is mapped to the graphics layer of the first display, and the first refresh bitmap data is transmitted to the graphics layer of the second display.

7. A display device, characterized in that: include: A first display is configured to display a drawing interface, wherein the drawing interface includes a display screen of a graphics layer; The first display is connected to a touch interaction module, and the touch interaction module is used to detect a touch track input by a user; when the touch track of an erase event input by the user is detected, the first display further displays the touch track of the erase event through an acceleration layer, and when the end point of the touch track is detected, the acceleration layer is released, and the display level of the acceleration layer when displaying the touch track is higher than the display level of the graphic layer; A device interface, configured to connect to a second display, the second display being used to display a display screen of the graphic layer of the first display; The controller is configured as: In response to a second touch track input by a user, detecting a bitmap background of a target graphic layer, wherein the second touch track is a touch track of an erase event, the second touch track includes at least one touch point, and the target graphic layer includes a graphic layer of the first display and a graphic layer of the second display; Detecting a second touch point and a second reference touch point of the second touch track, wherein the second touch point is a touch point with the latest touch time in the second touch track, and the second reference touch point is a touch point with the shortest touch time from the second touch point in the second touch track; Calculating an erasure area according to the second touch point and the second reference touch point; The bitmap background is refreshed to the acceleration layer and the graphics layer of the first display according to the erased area, and the bitmap background is transmitted to the graphics layer of the second display according to the erased area, so that the second display displays the same display screen as the first display.

8. The display device according to claim 7, characterized in that The controller is also configured to: In response to a start instruction of the annotation mode, taking a screenshot of the drawing interface to generate a background screenshot including the annotation view; The background screenshot is set as the bitmap background of the target graphics layer.

9. The display device according to claim 7, characterized in that: The controller is also configured to: Monitoring the touch points of the second touch track, wherein the touch points include a starting point, a track point, and an end point; When the end point of the second touch track is detected, the second target area is calculated according to the end point and the start point of the second touch track; The bitmap data of the target graphics layer is set as a transparent bitmap according to the second target area, and the bitmap data of the acceleration layer is released.

10. A multi-screen synchronous touch display method, characterized in that: Applied to the display device according to any one of claims 1 to 6, the method comprising: In response to a first touch track input by a user, generating target bitmap data of the first touch track, wherein the first touch track is a touch track of a drawing event and includes at least one touch point; Detecting a first touch point and a first reference touch point of the first touch track, wherein the first touch point is a touch point with the latest touch time in the first touch track, and the first reference touch point is a touch point with the shortest touch time from the first touch point in the first touch track; Calculating a drawing area according to the first touch point and the first reference touch point; The target bitmap data is refreshed to the acceleration layer and the graphics layer of the first display according to the drawing area, and the target bitmap data is transmitted to the graphics layer of the second display according to the drawing area, so that the second display displays the same display screen as the first display.

Images