Display device and device control method

Abstract

The invention provides a display device and a device control method, the method can respond to a window adjustment instruction of a first floating window, determine a first window area according to the window adjustment instruction, and obtain a window display area of a second floating window, and the second floating window is a floating window except the first floating window in a user interface. If the first window area and the window display area intersect, a second window area of a second floating window is calculated, and the second window area is used for representing the window display area which is not intersected with the first window area. And finally, displaying the first floating window in the first window area, and displaying the second floating window in the second window area. According to the method, when one floating window executes window adjustment, other floating windows can be subjected to self-adaptive adjustment, and non-overlapping display of the multiple floating windows is achieved.

Description

Display device and device control method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to the Chinese application No. 202311317445.8, filed on October 12, 2023; and No. 202311323158.8, filed on October 12, 2023, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a device control method. Background Art

[0004] A display device refers to a terminal device that can output a specific display screen, and can be a terminal device such as a smart TV, a communication terminal, or a smart advertising screen. Most current display devices have an application floating window function, which enables multiple applications to share the screen, allowing users to interact with multiple applications at the same time on the same screen, broadening the user's usage range and improving the user experience. During the interaction process, users can enlarge, reduce, move, and perform other operations on the application's floating window based on their needs. However, when multiple floating windows are displayed on the same screen, the adjustment of the position / size of one floating window will cause other floating windows to be covered, resulting in the inability to display the entire window.

[0005] Summary of the Invention

[0006] In a first aspect, an embodiment of the present disclosure provides a display device, comprising: a user input interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, the user input interface, the communication device and the memory, and configured to execute computer instructions to cause the display device to perform: in response to a window adjustment instruction for a first floating window, determining a first window area according to the window adjustment instruction; presenting at least two floating windows in the display, the first floating window being any one of the two floating windows; obtaining a window display area of ​​a second floating window, the second floating window being a floating window in the user interface other than the first floating window; if the first window area and the window display area intersect, calculating a second window area of ​​the second floating window, the second window area being used to represent a window display area that does not intersect with the first window area, the second window area being calculated based on the positional relationship between the first window area and the window display area; and controlling the display to display the first floating window in the first window area and the second floating window in the second window area.

[0007] In the second aspect, an embodiment of the present disclosure also provides a device control method, including: responding to a window adjustment instruction for a first floating window, determining a first window area according to the window adjustment instruction; presenting at least two floating windows in the display, the first floating window being any one of the two floating windows; obtaining a window display area of ​​a second floating window, the second floating window being a floating window other than the first floating window in the user interface of the display; if the first window area and the window display area intersect, calculating a second window area of ​​the second floating window, the second window area being used to represent a window display area that does not intersect with the first window area, the second window area being calculated based on the positional relationship between the first window area and the window display area; controlling the display to display the first floating window in the first window area, and to display the second floating window in the second window area. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to some embodiments;

[0009] FIG2 is a schematic diagram of a hardware configuration of a control device according to some embodiments;

[0010] FIG3 is a schematic diagram of a hardware configuration of a display device according to some embodiments;

[0011] FIG4 is a schematic diagram of software configuration of a display device according to some embodiments;

[0012] FIG5 is a schematic diagram of an interface of a floating window according to some embodiments;

[0013] FIG6 is a schematic diagram of scaling a floating window according to some embodiments;

[0014] FIG7 is a schematic diagram illustrating adjusting a floating window to cause overlapping display of floating windows according to some embodiments;

[0015] FIG8 is a schematic diagram illustrating overlapping display of floating windows caused by display of a newly triggered floating window according to some embodiments;

[0016] FIG9 is a schematic diagram of a process for adaptively adjusting multiple floating windows according to some embodiments;

[0017] FIG10 is a schematic diagram of an interactive area of ​​a floating window according to some embodiments;

[0018] FIG11 is a schematic diagram of the distance between floating window centers according to some embodiments;

[0019] FIG12 is a schematic diagram of a process for adaptively adjusting two floating windows according to some embodiments;

[0020] FIG13 is a schematic diagram of a process for adaptively adjusting three floating windows according to some embodiments;

[0021] FIG14 is a schematic diagram of a process for adjusting a second window area of ​​a window in a second window set according to some embodiments;

[0022] FIG15 is a schematic diagram of a display process of adaptive adjustment when a floating window is activated according to some embodiments;

[0023] FIG16 is a schematic diagram of a marking unit grid according to some embodiments;

[0024] FIG17 is a schematic diagram of a touch track according to some embodiments;

[0025] FIG18 is a schematic diagram of a whiteboard application interface according to some embodiments;

[0026] FIG19 is a schematic diagram of a whiteboard application display system according to some embodiments;

[0027] FIG20 is a schematic diagram of a note template for saving notes according to some embodiments;

[0028] FIG21 is a schematic diagram of a display device receiving and saving a newly created note element according to some embodiments;

[0029] FIG22 is a schematic diagram showing a display device storing note elements according to storage identifiers according to some embodiments;

[0030] FIG23 is a schematic diagram showing a display device determining a movement type of a sticky note element based on position change information according to some embodiments;

[0031] FIG24 is a schematic diagram showing a display device setting a zoom point according to a moving direction of a sticky note element according to some embodiments;

[0032] FIG25 is a schematic diagram showing a display device setting a zoom point according to a moving direction of a sticky note element according to some embodiments;

[0033] FIG26 is a schematic diagram of a display device adjusting template quadrant adaptive parameters according to some embodiments;

[0034] FIG27 is a schematic diagram illustrating a display device swapping positions of sticky notes within a template quadrant according to some embodiments;

[0035] FIG28 is a schematic diagram illustrating the intersection of a sticky note element and multiple quadrants according to some embodiments. DETAILED DESCRIPTION

[0036] The display device provided in the embodiments of the present disclosure can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figures 1 and 3 illustrate a specific embodiment of the display device of the present disclosure.

[0037] FIG1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. As shown in FIG1 , a user can operate a display device 200 through a smart device 300 or a control device 100 .

[0038] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0039] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0040] In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0041] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.

[0042] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0043] Figure 2 illustrates a block diagram of the configuration of a control device 100 according to an exemplary embodiment. As shown in Figure 2 , the control device 100 includes a processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0044] As shown in FIG3 , the display device 200 includes at least one of a tuner and demodulator 210 , a communication device 220 , a detector 230 , an external device interface 240 , a processor 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a communication external device interface.

[0045] In some embodiments, the processor includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0046] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0047] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0048] In some embodiments, the display device 200 may also support touch interaction, that is, a touch interaction module may be provided on the display device 200, which may be a built-in module or an external module. For example, to implement touch interaction operations, the display 260 of the display device 200 may be provided with a touch panel (TP), which can receive touch signals input by fingers, contacts, styluses, etc., and execute touch interaction responses according to pre-defined interaction strategies. Of course, the display device 200 may also be an integrated touch screen.

[0049] For another example, the display device 200 can also be connected to a touch interaction device (such as an air mouse), and the touch interaction device detects the touch signal input by the user to simulate a touch event, and sends the touch event to the display device 200, so that the display device 200 can complete different touch interaction operations according to the interaction strategy preset in the operating system.

[0050] Communication device 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communication device 220 to send and receive control signals and data signals with control device 100 or server 400.

[0051] The external device interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0052] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0053] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0054] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0055] In some embodiments, the processor 250 and the tuner 210 may be located in separate devices. Specifically, the tuner 210 may be located in a device external to the main device where the processor 250 resides, such as an external set-top box. The processor 250 controls the operation of the display device and responds to user operations using various software control programs stored in memory. The processor 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object for display on the display 260, the processor 250 may perform operations related to the object selected by the user command.

[0056] In some embodiments, the processor includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0057] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0058] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly format. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0059] As shown in Figure 4, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0060] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0061] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0062] As shown in Figure 4, the application framework layer in the embodiment of the present disclosure includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0063] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0064] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0065] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0066] Based on the above-mentioned display device 200, a specific screen can be presented. For example, a playback screen, a control interface, and other application interfaces. Depending on the developer or operator of the application, the application installed on the display device 200 may include various forms, such as system applications, third-party applications, etc. The applications installed on the display device 200 can also be divided into multiple types according to the functions that the application can achieve, such as video applications, reading applications, game applications, etc. When the application list is displayed on the display 260, the user can select one of the applications and control the display device 200 to run the application to achieve the functions of the application.

[0067] In some embodiments, the display device 200 has a floating window function and supports multi-window display. Multiple applications can be displayed on the screen in the form of floating windows, enabling multiple applications to be displayed on the same screen, broadening the user's usage range and improving the user experience. For example, as shown in Figure 5, the left side of the user interface displayed by the display device 200 displays the floating windows corresponding to application A and application B, so that the user can interact with application A and application B.

[0068] In addition, the startup logic of the floating window in the display device 200 is to start and display by switching from full screen to floating window, that is, the application starts in full screen mode and displays the application interface in full screen. When the floating small window instruction is received, the full screen mode of the application is switched to the floating window mode, forming a floating window displayed on the screen. After the display device 200 receives the floating small window instruction, it responds to the instruction to detect whether the FreeForm function is turned on. The FreeForm function refers to the multi-window display function in the display device 200 system. If it is not turned on, turn on the function. When the FreeForm function is turned on and available, obtain the small window size and window coordinates indicated in the floating small window instruction, and construct a floating window (ActivityOption) according to the small window size and window coordinates. After the floating window is constructed, start the small window floating window.

[0069] In some embodiments, a floating window includes two exit modes. The first mode is when a user inputs an exit command for the floating window to the display device 200. In response to the exit command, the display device 200 can cancel the display of the floating window and control the exit of the application corresponding to the floating window. The second mode is when the user inputs an exit command to the display device 200, the display device 200 can switch from the floating window mode to the full screen mode, returning the floating window to full screen display.

[0070] In some embodiments, for a floating window, a user can adjust the window based on needs, for example, to zoom in, zoom out, move, or other window adjustments. The user can control the display device 200 to perform corresponding window adjustment operations through specific interactive actions. For example, as shown in FIG6 , a schematic diagram of zooming in and out of a floating window is provided in an embodiment of the present disclosure. A touch screen (TP) is provided on the display 260 of the display device 200 to serve as a touch display. The user presses and keeps the finger in contact with the top corner of the floating window on the display 260 so that the display device 200 detects a touch press event (ACTION_DOWN). In response to the touch press event, the display 260 is controlled to display a window adjustment interface, i.e., a user interface in which the border of the floating window is highlighted. The user then keeps the finger in contact with the display 260 and moves it outward or inward so that the display device 200 detects a touch move event (ACTION_MOVE). In response to the touch move event, the floating window is zoomed in or out. When the user's finger leaves the display 260, the display device 200 detects a touch lift event (ACTION_UP). In response to the touch lift event, the zoomed in or out floating window is displayed.

[0071] In addition, to enhance the user experience, a window limit threshold can be set. When the user maintains contact with the display 260 and moves the finger outward, the display device 200 can detect the size of the floating window and directly execute full-screen operation when the window size exceeds the window limit threshold. For example, the default window size of the floating window when it is activated is 35% of the display area of ​​the display 260 screen. When the user controls the display device 200 to perform a window enlargement operation on the floating window through a specific interactive action, the display device 200 detects the size of the floating window. When the window size is enlarged to greater than 350% of the default window size, the window is executed full-screen, and the display 260 is controlled to display the floating window in full screen.

[0072] Before the window is transformed into a full screen, the window status may be displayed at the bottom of the user interface displayed on the display 260 to remind the user that the floating window is about to enter a full screen.

[0073] However, due to the display range limitation of the screen of the display device 200, when multiple floating windows are displayed on the screen of the display device 200, the position / size adjustment of a floating window may cause the floating window to cover other floating windows, thereby causing the multiple floating windows to be unable to be displayed completely.

[0074] For example, when multiple floating windows are displayed, one of the floating windows is adjusted so that the adjusted floating window covers the other floating windows. Taking the user interface shown in FIG5 as an example, the user inputs a window adjustment instruction to the display device 200 through a specific interactive action to enlarge the floating window of application A, so that the user interface presents the interface effect shown in FIG7, where the enlarged floating window of application A covers the floating window of application B.

[0075] For another example, when a floating window is activated, the newly triggered floating window will be displayed on a certain floating window, resulting in the phenomenon of multiple floating windows being superimposed in the user interface. Taking the user interface shown in Figure 5 as an example, the user enters the startup command of application C, and the application interface of application C is displayed in full screen. When a floating window instruction is received indicating that the full screen mode is switched to the floating window mode, the floating window of application C is displayed. The floating window of application C may be displayed on the floating window of application A and the floating window of application B, making the user interface present the interface effect shown in Figure 8.

[0076] Obviously, the above displays all require the user to manually shift or scale the covered floating window to achieve staggered display.

[0077] In order to solve the problem of floating windows being covered, some embodiments of the present disclosure also provide a device control method, which is applied to the display device 200, and is used to adaptively scale and / or position-switch the floating window to achieve non-overlapping display of multiple floating windows. Among them, the display device 200 to which the device control method can be applied includes: a display 260 and a processor 250. Among them, the display 260 is configured to display a user interface, and the user interface includes a floating window, and the processor 250 is configured to execute the above-mentioned device control method, including adaptive adjustment of multiple floating windows and adaptive adjustment when the floating window is started. As shown in Figure 9, Figure 9 is a schematic diagram of the process of adaptive adjustment of multiple floating windows in an embodiment of the present disclosure, which specifically includes the following contents:

[0078] S100: In response to a window adjustment instruction of a first floating window, determine a first window area according to the window adjustment instruction.

[0079] For adaptive adjustment of multiple floating windows, the display device 200 can control the display 260 to present at least two floating windows. By monitoring the floating windows displayed in the user interface, when a floating window performs window adjustment operations such as scaling and moving, it is determined whether the window display area after performing the window adjustment operation intersects (overlaps) with the window display area of ​​other floating windows. If so, the other floating windows are scaled and / or moved to achieve non-overlapping display of multiple floating windows.

[0080] Therefore, in order to make multiple floating windows fully displayed on the screen of the display device 200. The display device 200 can receive a window adjustment instruction of the first floating window input by the user during use, and in response to the window adjustment instruction, determine the first window area according to the window adjustment instruction, and judge whether the second floating window needs to be adjusted according to the first window area, and the second window area to which the second floating window should be adjusted when adjusting. Among them, the first floating window is a floating window that has been displayed in the user interface of the display device 200. The second floating window is a floating window other than the first floating window in the user interface of the display device 200. The first window area is the window display area after the first floating window is adjusted according to the window adjustment instruction, and the second window area is used to represent the window display area that does not intersect with the first window area.

[0081] In some embodiments, the display device 200 can obtain a floating window list through a database interface, wherein the floating window list includes currently displayed floating windows and related information thereof, including but not limited to application name, task identifier (taskid), and area coordinates.

[0082] For example, the display device 200 can query the application task list (task list) currently in the floating state, i.e., the floating window list, through the database interface "freepip_task_list_info" of the system global configuration (Settings.Global). The number of items in the floating window list can represent the number of floating windows displayed by the display device 200, and each item includes information such as the application name, task identifier (taskid), and area coordinates of the current floating window. In other words, the display device 200 can obtain information such as the window display area and coordinates of the floating window from the floating window list for adaptive adjustment of the floating window.

[0083] In some embodiments, the display device 200 can detect a window operation of a window adjustment instruction, wherein the window operation includes a move operation, a zoom-in operation, and a zoom-out operation. If the window operation is a move operation or a zoom-in operation, indicating that the second floating window needs to be adjusted, the subsequent step of determining the first window area according to the window adjustment instruction is performed. If the window operation is a zoom-out operation, indicating that the second floating window does not need to be adjusted, the first floating window is displayed in the first window area according to the window adjustment instruction.

[0084] That is, the display device 200 monitors the floating window, and when it monitors that the first floating window is a zoom-out operation, it does not process the second floating window. When it monitors that the first floating window is a zoom-in operation or a move operation, it further calculates the second window area of ​​the second floating window.

[0085] For the detection of window operations, the display device 200 can pre-set an interactive area for the first floating window, including a floating window zoom area and a floating window movement area. In response to the window adjustment instruction of the first floating window, the display device 200 can detect the starting coordinates in the window adjustment instruction, and determine the window operation of the window adjustment instruction based on the positional relationship between the starting coordinates and the first floating window interaction area. If the starting coordinates are within the preset floating window zoom area, the window operation of the window adjustment instruction is determined to be a zoom operation. If the starting coordinates are within the preset floating window movement area, the window operation of the window adjustment instruction is determined to be a movement operation.

[0086] As shown in Figure 10, a schematic diagram of the interactive area of ​​the floating window provided in an embodiment of the present disclosure is provided. The rectangular area on the upper side of the floating window is the floating window moving area, and the rectangular box on the outside is the floating window scaling area. For a display device 200 that supports touch function, when the user presses the screen of the display 260 with a finger, the display device 200 can monitor the touch press event (MotionEvent.ACTION_DOWN) and record the starting coordinates of the current touch point Point(x, y) = event.getX(), event.getY(). After that, the display device 200 will continue to track the position of the touch point, and when the user's finger leaves the screen of the display 260, a touch event ends. During the entire interaction, the display device 200 system distributes the MotionEvent object to the onTouchEvent() function, including relevant information for each interaction. For example, "MotionEvent.ACTION_DOWN" indicates that the first touch point is pressed, "MotionEvent.ACTION_UP" indicates that the last touch point leaves, and "MotionEvent.ACTION_MOVE" indicates that the touch point moves.

[0087] The display device 200 uses the relevant information provided by the MotionEvent object to determine whether a specific interactive action has occurred. Therefore, the display device 200 can obtain the touch start coordinates entered by the user's finger in the interface shown in Figure 8, and determine the current window operation based on the interactive area of ​​the floating window. When the touch start coordinates are within the floating window zoom area, the window operation is determined to be a floating window zoom operation. When the touch start coordinates are within the floating window movement area, the window operation is determined to be a floating window movement operation.

[0088] For the judgment of window scaling, the display device 200 can detect the starting coordinates and real-time coordinates in the window adjustment instruction, and obtain the center coordinates of the first floating window. Calculate the first distance between the center coordinates and the starting coordinates, and calculate the second distance between the center coordinates and the real-time coordinates. If the first distance is less than the second distance, mark the window operation of the window adjustment instruction as a magnification operation. If the first distance is greater than the second distance, mark the window operation of the window adjustment instruction as a reduction operation.

[0089] For example, taking the above touch event as an example, based on the "MotionEvent.ACTION_MOVE" callback, the display device 200 can obtain the updated touch coordinates UpdatePoint(x1, y1), that is, the real-time coordinates. Based on the current window display area of the first floating window, obtain the center coordinates CenterPoint(x0, y0).

[0090] Based on the Pythagorean theorem, calculate the distances between the center coordinates CenterPoint(x0, y0) and the starting coordinates Point(x, y), and the updated touch coordinates UpdatePoint(x1, y1) respectively according to the following formula: d1 = Math.sqrt[(x - x0) * (x - x0) + (y - y0) * (y - y0)]; d2 = Math.sqrt[(x1 - x0) * (x1 - x0) + (y1 - y0) * (y1 - y0)];

[0091] Among them, Math.sqrt() represents a function that returns the square root of a number, d1 represents the first distance, and d2 represents the second distance. If d1 < d2, that is, d2 - d1 > 0, it is a magnification operation. If d1 > d2, that is, d2 - d1 < 0, it is a reduction operation. Thus, the display device 200 performs the window scaling operation until it receives the touch release event (MotionEvent.ACTION_UP) and the instruction ends.

[0092] In addition, for the window scaling operation, the display device 200 can calculate the updated window boundary (mWindowDragBounds) when the floating window is dragged, and update the updated window boundary to the floating window list (Task list) through the resizeTask() method. Then, based on the floating window list, control the size and position of the image interface (surface) corresponding to the floating window, and notify SurfaceFlinger for update, where SurfaceFlinger is a system service in the display device 200 responsible for managing and drawing the screen image.

[0093] The first window area is determined according to the window adjustment instruction. The display device 200 can calculate the offset of the floating window based on the starting coordinate Point of the user's finger touch and the touch coordinate UpdatePoint after the finger slides and updates. Based on the offset, the floating window can be scaled or moved, and the new display position of the floating window, i.e., the first window area, can be calculated.

[0094] S200: Acquire a window display area of ​​a second floating window.

[0095] After obtaining the first window area indicated in the window adjustment instruction, the display device 200 can obtain the window display area of ​​the second floating window and determine whether the first window area of ​​the first floating window intersects with the window display area of ​​the second floating window. If they do not intersect, it means that the first floating window does not overlap with the second floating window after the window adjustment operation is performed, and the second floating window is not processed. If they do intersect, it means that the first floating window overlaps with the second floating window after the window adjustment operation is performed, and the second floating window needs to be adjusted to achieve non-overlapping display of multiple floating windows.

[0096] It should be noted that the present disclosure does not impose any specific limitations on the method for determining whether floating windows intersect, and those skilled in the art may adjust the method based on actual circumstances. For example, the method may be based on a basic coordinate algorithm (window four corner coordinate information) to determine whether two floating windows intersect, or the method may be based on the separating axis theorem to determine whether two floating windows intersect.

[0097] S300: If the first window area and the window display area intersect, calculate a second window area of ​​the second floating window.

[0098] The second window area is used to represent a window display area that does not intersect with the first window area, and the second window area is calculated based on the positional relationship between the first window area and the window display area.

[0099] In some embodiments, the second window area includes the window center coordinates and / or the window size, that is, when the first window area and the window display area intersect, the display device 200 can adjust the window center coordinates or the window size ratio of the second floating window until the first floating window and the second floating window do not intersect. When adjusting the second floating window, the position movement strategy is used first. The position movement strategy refers to adjusting the overall coordinate information of the second floating window. When the position movement strategy cannot meet the non-intersection rule, the window reduction strategy is used. The window reduction strategy means that the position of the window center point remains unchanged, and the window width and height are reduced.

[0100] Therefore, the display device 200 can calculate the positional relationship between the first window area and the window display area, where the positional relationship includes intersecting and non-intersecting. If the positional relationship is intersecting, the window display area is moved to obtain a second window area when the positional relationship is non-intersecting. If the second window area is not obtained, the window display area is reduced according to a preset reduction ratio, and then the second window area when the positional relationship is non-intersecting is obtained based on the reduced window display area.

[0101] In addition, to ensure the display clarity of the floating window, a window reduction threshold can be set. The window reduction threshold is the minimum window size of the window display area. If the window size of the reduced window display area is smaller than the window reduction threshold, the second floating window will not be processed. For example, if the window reduction threshold is 30%, the minimum window size of the window display area must not be smaller than 30% of the window size of the initial window display area.

[0102] In some embodiments, to improve the efficiency of window change execution, the second floating window can be determined based on the change in the distance between the center points of the two floating windows. The display device 200 can obtain the reference coordinates of the first floating window, the center coordinates of the first window area, and the center coordinates of the second floating window. The first relative distance between the reference coordinates of the first floating window and the center coordinates of the second floating window is calculated, and the second relative distance between the center coordinates of the first window area and the center coordinates of the second floating window is calculated. The reference coordinates are the center coordinates of the first floating window.

[0103] If the first relative distance is less than or equal to the second relative distance, it means that the first floating window is away from the second floating window after the window adjustment operation is performed, and the second floating window is not adjusted. The first floating window is adjusted to the first window area according to the window adjustment instruction. If the first relative distance is greater than the second relative distance, it means that the first floating window is close to the second floating window after the window adjustment operation is performed, and the step of calculating the second window area of ​​the second floating window is performed.

[0104] For example, as shown in Figure 11, which illustrates the distance between floating window centers according to an embodiment of the present disclosure, a first relative distance, dis1, is calculated using the Pythagorean theorem based on the center coordinates of the first floating window before the move / resize operation and the center coordinates of the second floating window. After the first floating window is moved / resized, its center coordinates and the center coordinates of the second floating window are obtained, and a second relative distance, dis2, is calculated. If dis2 ≥ dis1, indicating that the changed first floating window is farther away from the second floating window, no adjustment is made to the second floating window. Otherwise, the second floating window's second window area is calculated based on the intersection algorithm.

[0105] S400: Displaying a first floating window in a first window area, and displaying a second floating window in a second window area.

[0106] After calculating the second window area of ​​the second floating window, the display device 200 can display the first floating window adjustment in the first window area and display the second floating window in the second window area to complete the adaptive adjustment of the floating window. For example, the calculated second window area is the area value Rect(leftNew, topNew, rightNew, bottomNew), and the area value is used as the input parameter of the window adjustment (setLaunchBounds) to complete the adjustment of the display position of the second floating window.

[0107] FIG12 is a schematic diagram of a process for adaptively adjusting two floating windows according to an embodiment of the present disclosure. As shown in FIG12 , the process includes the following steps:

[0108] Step S1201: Read the area coordinates, window size and center point coordinates of two floating ports;

[0109] Step S1202: monitoring touch operations and changes in the floating window;

[0110] Step S1203: Determine whether there is a window adjustment instruction;

[0111] Display device 200 obtains a list of floating windows through a database interface and reads the region coordinates, window sizes, and center coordinates of the two floating windows. Display device 200 monitors touch operations and window changes of the floating windows to detect whether there are any window adjustment instructions. When a window adjustment instruction is detected for one of the floating windows, it is counted as the first window. The other floating window is counted as the second window.

[0112] Step S1204: If there is a window adjustment instruction, then detect the window operation of the window adjustment instruction; if there is no window adjustment instruction, then end the adjustment process;

[0113] Step S1205: If the window operation is a move operation or a zoom-in operation, then the adjusted first area of ​​the first window is obtained; if the window operation is a zoom-out operation, then the adjustment process is terminated;

[0114] Step S1206: determining whether the first window intersects with the second window;

[0115] Step S1207: If they intersect, calculate the second window area of ​​the second window;

[0116] Step S1208: If they do not intersect, adjusting the display position of the second window according to the second window area.

[0117] Detect the window operation of the window adjustment instruction. If the window operation is a zoom-out operation, end the adaptive adjustment step. If it is a move operation or a zoom-in operation, obtain the window position information of the first window adjusted according to the window adjustment instruction, that is, the first window area. Then, based on the intersection algorithm, calculate the position relationship between the adjusted first window and the second window. If the position relationship is an intersection, calculate the second window area of ​​the second window, including the window center coordinates and / or the window size ratio, until the two windows do not intersect. Finally, use the second window area as the input parameter for window adjustment, and adjust the display position of the second floating window according to the second window area to complete the adaptive adjustment of the floating window.

[0118] As further shown in FIG13 , FIG13 is a schematic diagram of a process for adaptively adjusting three floating windows provided in an embodiment of the present disclosure, including the following steps:

[0119] Step S1301: Read the area coordinates, window sizes, and center coordinate points of three floating windows;

[0120] Step S1302: monitoring touch operations and window changes of the floating window;

[0121] Display device 200 obtains a list of floating windows through a database interface and reads the region coordinates, window sizes, and center coordinates of each of the three floating windows. Display device 200 monitors touch operations and window changes of floating windows to detect whether there are any window adjustment instructions. When a window adjustment instruction is detected for one of the floating windows, it is counted as the first window. The other two floating windows are counted as the second window set, denoted as window A and window B, respectively.

[0122] Step S1303: Determine whether there is a window adjustment instruction;

[0123] Step S1304: If there is a window adjustment instruction, then detect the window operation of the window adjustment instruction; if there is no window adjustment instruction, then end the adjustment process;

[0124] Step S1305: If the window operation is a move operation or a zoom-in operation, then the adjusted first area of ​​the first window is obtained; if the window operation is a zoom-out operation, then the adjustment process is terminated;

[0125] Step S1306: determining whether the first window intersects with a window in the second window set;

[0126] Step S1307: If they intersect, then calculate the second window area of ​​the window in the second window set;

[0127] Step S1308: If they do not intersect, adjusting the display position of the window in the second window set according to the second window area.

[0128] The window operation of the window adjustment instruction is detected. If the window operation is a zoom operation, the adaptive adjustment step is terminated. If it is a move operation or a zoom operation, the window position information of the first window adjusted according to the window adjustment instruction, i.e., the first window area, is obtained. Then, based on the intersection algorithm, the positional relationship between the adjusted first window and the windows in the second window set is calculated. If the positional relationship is an intersection, the second window area of ​​window A and window B in the second window set is calculated. During the adjustment process, it is necessary to ensure that window A and window B do not intersect.

[0129] During adjustment, traverse window A and window B in the second window set and perform intersection judgment with the first window respectively to implement the non-overlap strategy. The implementation can be shown in Figure 14, including the following steps:

[0130] Step S1401: Acquire the adjusted first window area of ​​the first window;

[0131] Step S1402: Acquire the second window area of ​​window A;

[0132] Step S1403: Determine whether newDis1A is greater than or equal to dis1A;

[0133] Step S1404: if newDis1A<dis1A, then recalculate the second window area of ​​window A based on the intersection algorithm;

[0134] Based on the center coordinates of the first window before adjustment and the center coordinates of window A, the relative distance dis1A between the two windows is calculated using the Pythagorean theorem. After obtaining the first window area of ​​the adjusted first window, the center coordinates of the adjusted first window and the center coordinates of window A are obtained, and the relative distance newDis1A between the two windows is calculated. If newDis1A ≥ dis1A, the relative distance between the adjusted first window and window A remains unchanged or increases, and window A is not adjusted. Otherwise, the second window area of ​​window A is calculated using the intersection algorithm.

[0135] Step S1405: Acquire the second window area of ​​window B;

[0136] Step S1406: Determine whether newDis1B is greater than or equal to dis1B;

[0137] Step S1407: if newDis1B<dis1B, recalculate the second window area of ​​window B based on the intersection algorithm;

[0138] Similarly, based on the above logical algorithm, calculate the relative distance dis1B between the center coordinates of the first window before adjustment and the center coordinates of window B. Then calculate the relative distance newDis1B between the center coordinates of the adjusted first window and the center coordinates of window B. If newDis1B≥dis1B, it means that the relative distance between the adjusted first window and window B remains unchanged or increases, so window B does not need to be adjusted. Otherwise, based on the intersection algorithm, calculate the second window area of window B.

[0139] During the adjustment process, it is also necessary to ensure that window A and window B do not intersect. Calculate the center coordinate distance disAB between window A and window B before adjustment. Then, based on the second window areas of window A and window B after adjustment, calculate the center coordinate distance newDisAB between them. If newDisAB≥disAB, it means that the relative distance between the adjusted window A and window B remains unchanged or increases, so no adjustment is needed. If newDisAB<disAB, it means that the distance between the adjusted window A and window B is relatively close. Then, based on the intersection algorithm, determine whether they intersect. If they intersect, adjust the window with a larger distance change to achieve the adaptive adjustment of the floating window position in the second window set until they do not intersect.

[0140] Finally, based on the second window areas of window A and window B obtained in the above steps, use them as the input parameters for window adjustment respectively to complete the adaptive adjustment of the floating window.

[0141] The above examples are for the adaptive adjustment of two floating windows and three floating windows. The adaptive adjustment of multiple floating windows involves various window display modes such as double-window and triple-window. In actual application scenarios, multi-window display scenarios with four or more windows are used less frequently. Therefore, the display device 200 can only perform the adaptive adjustment of the floating window for the double-window and triple-window display scenarios. That is, the display device 200 can obtain the floating window list through the database interface. If the number of floating windows in the floating window list is 2 or 3, then execute the automatic adjustment of the floating window in the above example. If the number of floating windows in the floating window list is neither 2 nor 3, then do not execute the automatic adjustment of the floating window.

[0142] The above embodiment is the adaptive adjustment of multiple floating windows. The device control method provided by the embodiment of the present disclosure also includes the adaptive adjustment when the floating window is started. For the adaptive adjustment when the floating window is started, the display device 200 can filter and screen the current display area of ​​the user interface displayed by the display 260 in response to the startup instruction of the new floating window, calculate the target display area, the target display area is the floating window area that does not overlap with the displayed area, and create a new floating window according to the target display area. For example, the area value Rect(leftNew, topNew, rightNew, bottomNew) of the target display area is calculated as the input parameter of the window adjustment (setLaunchBounds), that is, ActivityOptions.setLaunchBounds(new Rect(leftNew, topNew, rightNew, bottomNew)), to complete the window display area setting of the new floating window.

[0143] FIG15 is a schematic diagram of a process for adaptively adjusting a floating window when it is activated according to an embodiment of the present disclosure. As shown in FIG15 , the process includes the following steps:

[0144] Step S1501: Obtain a floating window list;

[0145] Step S1502: Determine whether the number of windows in the floating window list is 0;

[0146] Step S1503: If the value is 0, a new floating window is created according to the preset default area;

[0147] Step S1504: If it is not 0, obtain the undisplayed area;

[0148] Step S1505: Locating the target display area within the non-display area based on the floating window area;

[0149] Step S1506: Create a new floating window according to the target display area; if the target display area is not located, jump to the aforementioned step S1503.

[0150] In response to the activation instruction for the new floating window, the display device 200 obtains a floating window list through a database interface. The display device 200 reads the window coordinates of the displayed floating window from the floating window list and calculates the displayed area based on the window coordinates. The displayed area is the area where the displayed floating window is displayed. The display device 200 then obtains the screen display area of ​​the display 260 and removes the displayed area from the screen display area to obtain the undisplayed area.

[0151] After calculating the undisplayed area, the target display area can be located within the undisplayed area based on the floating window area, and a new floating window can be created based on the target display area. The floating window area is the application display area of ​​the new floating window. For example, if the display aspect ratio of the application corresponding to the new floating window is 640 / 360, the aspect ratio of the floating window area is also 640 / 360. To better meet the personalized needs of users, the floating window area can also be pre-set according to user needs.

[0152] In some embodiments, as shown in FIG15 , after the display device 200 obtains the floating window list, it can read the number of windows in the floating window list that have displayed floating windows. If the number of windows is 0, that is, the floating window list is empty, indicating that no floating window is currently displayed, a new floating window can be created according to a preset default area. If the number of windows is not 0, indicating that a floating window is currently displayed, the displayed area can be removed from the screen display area of ​​the display 260 based on the current number of floating windows and their corresponding window display areas, and the target display area can be calculated from the remaining undisplayed area.

[0153] If the floating window list is empty, in order to improve the display effect, a new floating window can be displayed in the center area of ​​the screen of display 260, that is, the preset default area is the center area of ​​the screen of display 260. The default area can be calculated based on the size of the screen display area of ​​display 260 and the size of the floating window area.

[0154] For example, if the display aspect ratio of the application corresponding to the new floating window is 640 / 360, the height of the default area can be the height of the screen display size of the display 260, and the width of the default area can be obtained by conversion based on the aspect ratio. That is, the area values ​​(left, top, right, bottom) of the default area can be calculated according to the following formula: left = screenWidth / 2 - freeformWidth / 2; top = screenHeight / 2 - freeformHeight / 2; right = screenWidth / 2 + freeformWidth / 2; bottom = screenHeight / 2 + freeformHeight / 2;

[0155] Among them, freeformWidth and freeformHeight represent the height and width of the floating window area, which are 640 and 360 respectively. screenWidth and screenHeight represent the height and width of the screen display area.

[0156] In some embodiments, when locating the target display area within the undisplayed area based on the floating window area, the display device 200 may divide the screen display area into multiple unit grids, traverse the multiple unit grids, determine the inclusion relationship between each unit grid and the displayed area, mark the unit grids outside the displayed area with a first identification grid, and mark the unit grids within the displayed area with a second identification grid. The target display area is located within the undisplayed area based on the floating window area. The target display area is the display area when all the unit grids covered by the floating window area are the first identification grids.

[0157] In some embodiments, as shown in FIG16 , FIG16 is a schematic diagram of marking a unit grid in an embodiment of the present disclosure. The display device 200 currently displays two floating windows, namely, floating window 1 and floating window 2. When triggering the startup of a new floating window, the display device 200 constructs a rectangle based on the screen width and height of the display 260, and divides the rectangle into multiple unit grids. Traverse all unit grids, determine the inclusion relationship between each unit grid and the window display area corresponding to floating window 1 and floating window 2, and if the unit grid is outside the window display area corresponding to floating window 1 and floating window 2, it is marked as 1. If the unit grid is within the window display area corresponding to floating window 1 or floating window 2, it is marked as 0. Use the floating window area as a sliding window to slide in the area composed of 1, and traverse the area composed of 1 in turn. When the area covered by the floating window area is marked as 1, it is marked as the target display area, and its starting coordinates are recorded as the starting coordinates of the new floating window.

[0158] In some embodiments, to make the display area of ​​the floating window more compatible with human visual perception, the display device 200 sets an expansion mechanism for the center point of the display 260 screen when locating the target display area. That is, when locating multiple target display areas, the target display area with the center of the display 260 screen as the expansion baseline is preferentially selected.

[0159] Therefore, the display device 200 can obtain the center coordinates of the target display area and the center coordinates of the screen display area, calculate the center distance between the center coordinates of the target display area and the center coordinates of the screen display area, determine the adapted display area based on the center distance, and create a new floating window in the adapted display area. The adapted display area is the target display area whose center distance is less than a preset center threshold.

[0160] The display device 200 may also obtain a target display area list, which includes multiple located target display areas. The target display areas in the target display area list are traversed, and the center coordinates of each target display area are calculated. The center distance between the center coordinates of each target display area and the center coordinates of the screen display area is then calculated. The target display area with the smallest center distance, i.e., the target display area closest to the center of the display 260 screen, is selected as the adapted display area.

[0161] It should be noted that the embodiment of the present disclosure does not specifically limit the calculation method of the center distance, and those skilled in the art can adjust it according to actual conditions. For example, the distance between two points can be calculated based on the Euclidean distance formula.

[0162] In some embodiments, when the display device 200 fails to locate the target display area, that is, when the target display area list is empty, the floating window area can be scaled according to a preset ratio, and the target display area can be located within the undisplayed area based on the scaled floating window area. In other words, when the target display area is not located, the height and width of the floating window area can be adjusted to reduce the display size ratio, and then the target display area list can be obtained according to the above-mentioned method for locating the target display area, and the adaptive display area can be selected.

[0163] In addition, to ensure the display clarity of the floating window, a window limit value can be set. The window limit value is the minimum window size of the floating window area. If the window size of the reduced floating window area is smaller than the window limit value, a new floating window is created according to the preset default area. For example, the window limit value is 30%, that is, the minimum window size of the floating window area cannot be less than 30% of the initial floating window area size. If it is smaller than the window limit value, the content of the floating window will be unclear, which is not conducive to user use.

[0164] Based on the above device control method, some embodiments of the present disclosure further provide a display device 200, including: a display 260 and a processor 250. The display 260 is configured to display a user interface, wherein the user interface includes a floating window; the processor 250 is configured to execute the following program steps:

[0165] S100: In response to a window adjustment instruction of a first floating window, determine a first window area according to the window adjustment instruction.

[0166] S200: Acquire a window display area of ​​a second floating window.

[0167] The second floating window is a floating window other than the first floating window in the user interface.

[0168] S300: If the first window area and the window display area intersect, calculate a second window area of ​​the second floating window.

[0169] The second window area is used to represent a window display area that does not intersect with the first window area, and the second window area is calculated based on the positional relationship between the first window area and the window display area.

[0170] S400: Displaying a first floating window in a first window area, and displaying a second floating window in a second window area.

[0171] It can be seen from the above technical solution that the display device and device control method provided in the above embodiment can respond to the window adjustment instruction of the first floating window, determine the first window area according to the window adjustment instruction, and obtain the window display area of ​​the second floating window, wherein the second floating window is a floating window other than the first floating window in the user interface. If the first window area and the window display area intersect, the second window area of ​​the second floating window is calculated, wherein the second window area is used to represent the window display area that does not intersect with the first window area. Finally, the first floating window is displayed in the first window area, and the second floating window is displayed in the second window area. The method can adaptively adjust other floating windows when one floating window performs window adjustment, so as to achieve non-overlapping display of multiple floating windows.

[0172] Considering that in real-world applications, a common scenario for users to use multi-window display on display devices is to browse notes. Specifically, users can use notes to record information while using the display device, so that they can be easily accessed during subsequent use. The type and format of information to be recorded may vary, so the note editor can be used to edit notes in multiple formats to suit various recording needs.

[0173] Most display devices use a note editor for note editing, and the notes generated by the note editor will be displayed in disorder on the screen, making the display interface of the display device cluttered. Although the current note editor can arrange the notes in order on the display interface after they are generated. However, only displaying the generated notes in an orderly manner still cannot clearly classify the information recorded by the user, which is not conducive to the user's subsequent search during use. In addition, the related art also saves the notes generated by the note editor by setting a label template to facilitate user search. However, when the number of notes stored in the label template is large, it is easy to cause the display effect of each note to be poor, so that the user needs to spend more time when searching for notes.

[0174] Based on this, the display device 200 of the embodiment of the present disclosure is also provided with a touch interaction module, which can receive touch data input by the user through the touch interaction module. According to the interaction principle of the touch interaction module, the touch interaction module can detect relevant parameters in the touch action. For example, the display 260 of the display device 200 is provided with a capacitive touch layer. When the user's finger touches any position of the touch layer, the capacitance of the touch layer at the touch position will change. At this time, the touch interaction module can record the position of the capacitance change and record the time when the capacitance change occurs in combination with the timer in the processor to generate touch data.

[0175] In some embodiments, the physical signals detected by the touch interaction module need to be converted before they can be read by the processor 250 of the display device 200. For example, an analog signal of capacitance change needs to be converted into a digital signal before being transmitted to the processor 250 of the display device 200. To adapt to the screen display process of the display device 200, the touch data detected by the touch interaction module also needs to be coordinate mapped so that the location corresponding to the touch data can correspond to the pixel points in the display 260 of the display device 200 or the user interface. For example, if the coordinate position detection accuracy of the touch interaction module is 65536×65536, and the resolution of the display 260 of the display device 200 is 3840×2160, then after the touch interaction module detects the touch data, the touch point position in the width and height directions can be mapped according to the ratio of 65536 / 3840 and 65536 / 2160, respectively, so that the display device 200 can perform relevant operation responses based on the mapped touch point position coordinates.

[0176] Among them, the touch data can include different operation parameters according to the user's input method. For example, as shown in Figure 17, the touch data may include the touch time, touch position, touch event type, number of touch points, etc. of the touch operation. During a touch interaction process, the touch event types that can be detected by the touch interaction module include: down events, move events, and up events of touch actions. Corresponding to the entire touch trajectory, a starting point [down; x1, y1, t1], a trajectory point [move; x2, y2, t2], and an end point [up; x3, y3, t3] can be formed. Each touch event records the corresponding touch time t and touch position (x, y).

[0177] It should be noted that, since the user's finger or touch device is in surface contact with the touch interaction module when performing touch interaction operations, that is, the touch interaction module can detect a contact area, which includes multiple contact points. Therefore, when the display device 200 performs interactive control, it can detect (or specify) a contact point in the contact area as a touch point. For example, when the contact area is an elliptical area formed when the user's finger contacts the touch interaction module, the display device 200 can traverse the contact point coordinates in the elliptical area to determine the coordinate extremes in two mutually perpendicular directions to delineate the major axis and minor axis of the elliptical area, and use the contact point at the intersection of the major axis and the minor axis as the touch point.

[0178] In the process of executing the touch interaction response, the display device 200 can classify different touch times, touch positions, number of touch points and touch event types, so as to detect the user's touch input gesture, thereby performing different interactive responses for different input gestures. For example, when the touch data detected by the touch interaction module has a touch point number of 1, the touch event type only includes a down event and an up event, and the time interval between the down event and the up event is less than the single-click event time threshold. Moreover, if the touch position change distance of the down event and the up event is less than the jitter distance threshold, it can be determined that the current user's touch input gesture is a click gesture. Based on the click gesture, the display device 200 executes the operation of starting the application according to the icon in the area where the touch position corresponding to the click gesture is located.

[0179] Similarly, the display device 200 can also detect other touch interaction gestures input by the user based on the touch duration, touch location, number of touch points, and touch event type, such as a double-click gesture, a slide gesture, a long press gesture, a multi-finger click gesture, and a multi-finger slide gesture. Different interaction gestures can derive different touch parameters. For example, a slide gesture can also generate parameters such as slide distance, slide speed, slide direction, and slide trajectory shape. Different touch parameters can be used to perform different interactive controls.

[0180] In some embodiments, in order to enrich the touch interaction types supported by the display device 200, the display device 200 can also perform complex gesture detection based on the dynamic changes in touch time, touch position, number of touch points and touch event type. For example, when the touch interaction module detects that the touch data input by the user includes multiple touch points, and the touch position of each touch point is close to each other as the user inputs, it can be determined that the touch gesture currently input by the user is a grabbing gesture. Based on complex gesture detection, the display device 200 can perform more interactive responses. For example, when the detected touch gesture is a grabbing operation for an image view, the display device 200 can respond to the grabbing gesture and perform a control action to select and move the image display position.

[0181] In some embodiments, touch gestures can also be combined with touch positions to form touch interaction operations based on specific positions. For example, when the touch interaction module detects that the down event position during the touch operation is at the top of the display 260, the touch point position corresponding to the move event moves downward. When the up event is detected, the display device 200 can determine that the current touch gesture is a swiping down from the top of the screen. Therefore, the display device 200 can control the display 260 to present a drop-down list in response to the touch gesture.

[0182] Some touch interaction gestures can be used to perform interactive actions based on the operating system of the display device 200, and are referred to as global gestures. For example, swiping down from the top of the screen in the example above on the display device 200 to bring up a drop-down list. Some touch interaction gestures can be used to perform interactive actions based on specific touch applications. For example, when the display device 200 is running a whiteboard application, a user-inputted slide touch action can be used to draw a track line on the whiteboard.

[0183] Touch applications can be system applications or third-party applications developed based on the specific display device type and user needs. Touch applications can be installed on the display device 200 and launched and run under user control. For example, a user can control the display device 200 to run the whiteboard application by clicking the "whiteboard" application icon in the application interface of the display device 200.

[0184] It should be noted that a whiteboard application can be a standalone application, an associated application, or a functional module within a touch application. Different whiteboard applications can have different startup or invocation methods. For example, a user can launch a standalone whiteboard application through the application interface of the display device 200; a user can also launch a linked whiteboard application by performing a "whiteboard" operation on a conference application; and a user can control the conference application to launch the whiteboard functional module by clicking the "whiteboard" icon on the conference application.

[0185] Different forms of whiteboard applications may present different whiteboard application interface forms after startup. For example, when a user starts a whiteboard application through an associated application form, the display device 200 may automatically enter dual-screen (or split-screen) mode. In dual-screen (or split-screen) mode, the display device 200 may display the user interface through two display areas, i.e., the main screen displays the original application (such as a conference application) interface, and the secondary screen displays the whiteboard application interface.

[0186] After the whiteboard application is running, the display device 200 can switch to displaying a whiteboard interface. As shown in FIG18 , the whiteboard interface may include drawing tool controls for drawing operations, such as pen shape, graphics, line shape, color, erase, clear screen, and selection. Users can click any control to draw graphics according to their specific drawing needs.

[0187] As shown in FIG19 , the display device 200 can render and display the drawn picture in real time through an image generation (Image Producer) module, an image rendering (surface flinger) module, a synthesis (Composer) module, a video processing unit hardware (VPU hardware) module, and a display (screen).

[0188] During the display of the drawn image, the graphics interface (OpenGLES) of the image generation module can be connected to the whiteboard application to generate the drawn image content and pass the image content to the image rendering module. The image rendering module is used to control the display content of the display 260, that is, to render the image content through the user interaction layer (UI layer) or the video layer according to the image content. To achieve a better display effect, the image content of the UI layer can be rendered by the GPU component, thereby forming a picture frame for display in the synthesis module. Among them, the synthesis module can store the picture frames through the frame buffer and video buffer respectively, and the video processing module forms the picture content in the corresponding layers. That is, the image frames generated during the video playback process form the picture content through the video plane, while the image frames of other image screens (such as touch tracks, graphics, etc.) form the picture content through the primary OSD plane. Finally, the display device 200 forms the final picture through the blending unit (blend) of the video processing module and displays it on the display 260.

[0189] Based on the software and hardware architectures shown in Figures 1-19 , a note editor can be loaded into the display device 200 to facilitate users to temporarily record required information. Alternatively, whiteboard software can be loaded into the display device 200 and a note editor can be embedded within the whiteboard software. Users can use the note editor to create notes and record required information on the notes.

[0190] For the generated sticky notes, the user can adjust the position of the generated notes by means including but not limited to touch operations to store the generated notes in the expected position. The display device 200 can also store the notes in a suitable sticky note template according to the identifier of the generated note. It should be noted that after the display device 200 automatically stores the notes in a suitable sticky note template, the user can still change the storage location of the notes by means of touch operations. It is understandable that when changing the storage location of the notes by touch operations, the technical solution of the touch interaction action combined with touch gestures and touch position recognition in the above-mentioned embodiment can be used to respond to the user's move instruction to move the notes by touch.

[0191] When generating notes, the format of the notes generated by the note editor can also be different depending on the information the user needs to record. The note format can be selected to adapt to the type of information to improve the practicality of using notes to record information. When the number of notes is large, a note template can be set to store the notes. The note template can include multiple template quadrants, each template quadrant corresponds to a note format, and then the notes can be saved in categories for easy viewing by the user.

[0192] In the template quadrant, sticky notes will still be displayed at a certain scale ratio to facilitate user viewing and selection of sticky notes. It is understandable that when there are a large number of sticky notes, the size of the sticky notes needs to be reduced proportionally and displayed according to the reduced size; when the number of sticky notes is small, the sticky notes can be displayed according to the original size of the sticky notes or proportionally enlarged. However, as shown in Figure 20, when the user calls the sticky notes in the template quadrant, the number of sticky notes in the template quadrant changes due to operations such as moving the sticky notes into or out of the template. At this time, the size of the sticky notes is still adjusted and displayed according to the inherent proportional adjustment method, which is not conducive to the use of the display space used to display sticky notes in the template quadrant, resulting in the user taking a long time to find the target sticky notes.

[0193] As shown in FIG21 , at least one processor in the display device according to the embodiment of the present disclosure is further configured to execute computer instructions so that the display device performs the following steps:

[0194] S2101: In response to an operation instruction for creating a new note element, receiving a note element;

[0195] In some embodiments, the operation instruction may refer to a user creating a new note element through the note editor. At least one processor in the display device detects the newly created note element, receives the note element, and stores the note element in a note template. The note template includes multiple template quadrants, as shown in Figure 22. Each template quadrant can represent a different note format, thereby categorizing the note elements to facilitate the user's subsequent use of the note according to their needs.

[0196] Operation instructions can be input by the user to the note editor or display device 200 through various input methods such as voice, gestures, and touchpad. The display device interacts with the user in various ways, which can be adapted to various application scenarios and provide convenience for the user. Among them, the voice input form can be combined with the external device interface 240 of the display device 200 to connect to a sound acquisition device, and combined with the built-in audio processor to recognize the user's instructions. It is also possible to connect an image acquisition device through the external device interface 240 to collect the user's gestures and analyze them to interpret the user's operation instructions.

[0197] It is understood that each template quadrant includes a title area, a content area, and a footer area, and the shape, color, and text of each area can be freely adjusted. In some embodiments, the shape of each area can be adaptively adjusted based on the sticky note elements stored in the template quadrant to adjust the display mode of the sticky note elements to the optimal state for user search. Users can also modify the text and graphics in the title area and footer area that serve as prompts to facilitate subsequent search.

[0198] In some embodiments, when a user creates a new note element through the note editor, the display interface of the display device 200 may already display the note element. At this time, the newly created note element and the displayed note element are displayed at the same time, which may easily cause mutual obstruction. Therefore, when at least one processor in the display device detects a newly created note element, it can obtain the display content of the current display interface and use it to process the newly created note element. That is, at least one processor is specifically configured to execute computer instructions so that the display device receives the note element by executing the following in response to the operation instruction for creating a new note element: if it is detected that the display interface of the display includes a note template, then the step of storing the note element to the template quadrant of the note template according to the storage identifier of the note element is executed; if it is detected that the display interface of the display does not include a note template, then the display is controlled to display the received note element.

[0199] In some embodiments, when at least one processor in the display device detects that a note template is already displayed on the current display interface, it can be determined that the user needs to use the classification function of the note template, that is, the note element can be stored in the template quadrant of the note template according to the storage identifier of the newly created note element. Each note element has a storage identifier, and the storage identifier can be the color of the note element. It is understandable that the template quadrants used to store different types of notes can be distinguished by different colors. Then, at least one processor in the display device can store the note element in the template quadrant with the same color as the note element according to its color.

[0200] Storing note elements in template quadrants based on storage identifiers facilitates quick classification of notes and makes them easier for users to find. Furthermore, using color as a storage identifier not only allows for categorized storage but also makes it easier for users to find the desired note type based on color.

[0201] In some embodiments, if at least one processor in the display device detects that the current display interface does not display a note template, it can be determined that the user does not currently need to use the note template classification function. When a user creates a new note using a note editor in a mobile terminal such as a mobile phone, if a template quadrant associated with the user's new note exists on the display device, the new note needs to be stored in the associated template quadrant based on the storage identifier carried by the user's new note, thereby achieving classification statistics. For example, in a meeting, it can be used to classify and count user opinions.

[0202] It should be noted that for note elements that are not automatically stored in the associated template quadrant by the display device, they can also be moved to the template quadrant through user input. For example, when a note element is generated on the display device, the note template is not displayed, but the user can activate the note template and manually drag and drop the note element to the note template.

[0203] S2102: storing the note element in a template quadrant of the note template according to the storage identifier of the note element;

[0204] In addition to using the color / background color of the note element as a storage identifier as mentioned in the above embodiment, the key characters, shape, etc. of the note element can also be used as a storage identifier. In some embodiments, the template quadrant in the note template has a certain capacity limit. Too many note elements are not conducive to display, which in turn affects the user's search speed. In addition, for temporarily created note elements, they may not belong to any template quadrant in the note template. Therefore, a non-template quadrant can be set in the note template to increase the storage capacity of the note template. That is, at least one processor is specifically configured to execute computer instructions to cause the display device to store the note element in the template quadrant of the note template according to the storage identifier of the note element by performing the following operations: if the number of note elements stored in the template quadrant is greater than the storage number threshold, the note element is stored in the non-template quadrant of the note template; if the number of note elements stored in the template quadrant is less than or equal to the storage number threshold, the note element is stored in the template quadrant of the note template according to the storage identifier of the note element.

[0205] The storage quantity threshold is the maximum value of the note elements that can be stored in each template quadrant. When storing a note element, at least one processor in the display device can obtain the storage quantity threshold of the template quadrant associated with the newly created note element and the number of currently stored note elements, and then determine whether to save the newly created note element to the template quadrant.

[0206] In some embodiments, after at least one processor in the display device obtains the storage quantity threshold of the template quadrant and the number of currently stored note elements, if it is found through comparison that the number of currently stored note elements in the template quadrant is less than the storage quantity threshold, the note element can be stored in the associated template quadrant according to the storage identifier of the newly created note element.

[0207] In some embodiments, at least one processor in the display device determines, through comparison, that the number of note elements currently stored in the template quadrant is greater than a storage quantity threshold, and then the newly created note element can be stored in a non-template quadrant. When the storage quantity of the template response associated with the storage identifier of the note element reaches the maximum value, at least one processor in the display device controls the display to display a prompt message on the display interface to prompt the user that the storage quantity of the current template quadrant has reached the maximum value. Furthermore, when displaying the prompt message, other template quadrants can be provided to the user for storing note elements, or the user can be prompted that the note element is about to be stored in a non-template quadrant.

[0208] By setting a storage quantity threshold for the template quadrant, the problem of difficulty in user search caused by too many note elements stored in the template quadrant can be alleviated. In addition, when the number of any one type of note elements is large, multiple template quadrants of the same type can be set according to the storage identifiers of the larger number of note elements. When storing note elements, at least one processor in the display device stores note elements with the same storage identifier in multiple template quadrants with the same storage identifier to facilitate user search. Multiple template quadrants can simultaneously view multiple note elements with the same storage identifier when the user searches for a note element, thereby improving search efficiency.

[0209] It is understandable that at least one processor in the display device can adopt a dynamic monitoring method for the note template to flexibly adjust the note elements in each template quadrant. For example, the newly created note element A is saved in the non-template quadrant because the number of note elements stored in the template quadrant has reached the maximum value. However, in the process of using the note, the user deletes some note elements in the template quadrant, and then the storage quantity in the template quadrant associated with the note element A decreases, which can be used to store the note element A. At least one processor in the display device can move the note element A from the non-template quadrant to the template quadrant associated with the note element A. The note elements in the template quadrant can be flexibly adjusted through dynamic monitoring to improve the classification ability of the note elements.

[0210] S2103: adjusting size parameters of the sticky note elements in the template quadrant according to the number of the sticky note elements in the template quadrant;

[0211] Newly created sticky note elements have original size parameters, and display 260 may display the newly created sticky note elements according to the original size parameters. It is understood that the original size parameters may be the user's ideal size parameters. A sticky note template is used to store sticky note elements within a template quadrant. Therefore, when a sticky note element is stored within a template quadrant, it must be scaled according to the original size parameters to facilitate storage within the template quadrant.

[0212] Each template quadrant can directly display some of the stored note elements, and the display size of the note elements displayed in the template quadrant is the same. Therefore, the size parameters of the note elements in the template quadrant can be adjusted according to the number of note elements in the template quadrant to serve as the size parameters displayed in the template quadrant. That is, at least one processor is specifically configured to execute computer instructions so that the display device adjusts the size parameters of the note elements in the template quadrant according to the number of note elements stored in the template quadrant by performing the following: calculating the adaptive storage parameters of the template quadrant according to the number of note elements stored in the template quadrant, the adaptive storage parameters including note row spacing, note column spacing, and edge spacing; adjusting the size parameters of the note elements in the template quadrant based on the adaptive storage parameters; and controlling the display to display the note elements in the template quadrant.

[0213] Adaptive storage parameters are dynamically adjusted by at least one processor in the display device based on the number of note elements within the template quadrant to optimize the display of the note elements displayed within the template quadrant. Adaptive parameters may include note row spacing, note column spacing, and edge spacing. Edge spacing refers to the distance between a note element and the border of the template quadrant, while note row spacing and note column spacing refer to the distance between the borders of the note elements. By adjusting the adaptive storage parameters, the size parameters of the note elements displayed within the template quadrant can be adjusted.

[0214] It should be noted that the sticky note element whose display size is adjusted within the template quadrant will still carry its original size parameters for display on the display interface of display 260 when called. The sticky note element stored in the template quadrant will also carry the quadrant template ID and quadrant index, which can be used to determine the movement method of the sticky note element when it is moved. The sticky note element can be classified into the new template quadrant according to the user's instructions.

[0215] S2104: In response to an operation instruction for moving a sticky note element, acquiring position change information of the sticky note element;

[0216] At least one processor in the display device 200 can establish a coordinate system based on the display interface of the display device 200. The position of the newly created sticky note element can be described by coordinates, and the position of the sticky note template and the template quadrant can also be described by coordinates. When a user moves a sticky note element, for example, the sticky note element is in the shape of a rectangle, the vertex coordinates of the sticky note element can be obtained. The vertex coordinates will change before and after the sticky note element is moved. The change in the vertex coordinates can be regarded as the position transformation information of the sticky note element. The position transformation information can be used to determine the position of the moved sticky note, and the ownership of the moved sticky note can be determined based on the position of the moved sticky note.

[0217] In some embodiments, at least one processor is specifically configured to execute computer instructions so that the display device obtains the position transformation information of the sticky note element in response to the operation instruction for moving the sticky note element by executing the following: if the display interface of the display includes a sticky note element and a sticky note template, then the movement type of the movement operation is parsed, and the movement type is used to characterize the relative movement between the sticky note element and the sticky note template, and the movement type includes sticky note moving in, sticky note moving out, sticky note quadrant transformation, and intra-quadrant transformation; the sticky note template and sticky note element can be displayed on the display interface, and when the sticky note template is displayed, some sticky note elements will also be displayed in each template quadrant. Therefore, when receiving the user's operation instruction to move the sticky note element, it is necessary to determine the sticky note element to be moved and the relationship between the moved sticky note element and the sticky note template, and then adjust the sticky note element according to the user's operation instruction.

[0218] Moving a note in refers to the process of moving a note element outside the note template into the note template to store it. Moving a note out refers to the process of moving a note element inside the note template to display it at its original size. Transitioning between quadrants refers to the process of moving a note element from one template quadrant to another. Transitioning within a quadrant refers to adjusting the position of a note element within the current quadrant.

[0219] In some embodiments, at least one processor in the display device determines, based on the position transformation information of the sticky note element, that the sticky note element movement type corresponding to the user's operation instruction is a sticky note move-in. If the movement type is a sticky note move-in, the sticky note element is stored in a template quadrant that intersects with the sticky note element.

[0220] It should be noted that when a sticky note is moved, the sticky note element may not intersect any template quadrant. In this case, the at least one processor in the display device can place the sticky note element in a non-template quadrant to store the sticky note element. Alternatively, the at least one processor in the display device can dynamically adjust the sticky note template to adjust the sticky note element from the non-template quadrant to the template quadrant associated with the storage identifier of the sticky note element.

[0221] In some embodiments, at least one processor in a display device can determine whether a note element has been stored in a template quadrant based on its quadrant index. It should be understood that only note elements stored in a template quadrant carry the quadrant index, and the note element also carries its specific position information in the template quadrant. Furthermore, the at least one processor in the display device can determine the type of movement of the note element based on the quadrant index and the position transformation information of the note element, and move the note element. The at least one processor is specifically configured to execute computer instructions so that the display device, in response to an instruction to move the note element, obtains the position transformation information of the note element by: before performing a move operation on the note element, reading the quadrant index of the note element; if the note element is not stored in the note template, obtaining the position transformation information of the note element when performing the move operation, the position transformation information including the amount of movement of the note element in a first direction and a second direction; and if the note element intersects a template quadrant in the note template, marking the movement type as a note move-in.

[0222] Before moving a sticky note element in response to a user's command, at least one processor in the display device reads the quadrant index of the sticky note element to be moved. If the sticky note element carries the quadrant index, the sticky note element is considered to have been stored in the template quadrant. Therefore, when moving the sticky note element into the template quadrant, the number of sticky note elements in the template quadrant does not need to be changed. Instead, the existing sticky note elements in the template quadrant need only be updated based on the moved sticky note element / the content recorded by the moved sticky note element.

[0223] If the note element does not carry a quadrant index, the current move operation of the note element is considered a storage operation. At the same time, the template quadrant intersecting with the note element can be determined based on the position transformation information of the note element, so that the note element can be stored in the template quadrant according to the user's operation instruction. The first direction movement amount in the position transformation information can refer to the movement amount in the x-axis direction in the coordinate system established based on the display interface, and the second direction movement amount can refer to the movement amount in the y-axis direction in the coordinate system established based on the display interface.

[0224] Based on the coordinates of the note element before it was moved, combined with the amount of movement in the first direction, the amount of movement in the second direction, and the coordinates of the template quadrant, the template quadrant that intersects with the note element can be determined, and the note element can be stored in the template quadrant. In other embodiments, a note element can be removed from a template quadrant based on a similar determination method. For the removed note element, at least one processor in the display device can display the removed note element based on the original size parameters of the note element. That is, if the movement type is a note removal, the note element is displayed based on the initial size parameters of the note element. That is, the at least one processor is specifically configured to execute computer instructions to cause the display device to adjust the display position of the note element based on the position transformation information by performing the following operations: before performing a move operation on the note element, reading the quadrant index of the note element; if the note element is within the template quadrant, obtaining the position transformation information of the note element when the move operation is performed; and if the note element does not intersect with the template quadrant in the note template, marking the movement type as a note removal.

[0225] At least one processor in the display device can determine that the note element has been stored in the template quadrant based on the quadrant index of the note element. The processor 250 can also determine that the note element is located within the template quadrant based on the current coordinates of the note element and the coordinates of the template quadrant. As shown in Figure 23, in the same coordinate system established based on the display interface, the template quadrant is a rectangle, and its four vertex coordinates are (2, 1), (2, 9), (8, 1), (8, 9); the four vertex coordinates of the note element are (3, 4), (3, 6), (5, 4), (5, 6); then the rectangle formed by the vertex coordinates of the note element is within the rectangle formed by the vertex coordinates of the template quadrant, and the at least one processor in the display device can determine that the current note element is located within the template quadrant.

[0226] At least one processor in the display device first determines the position of the note element before it moves, and then determines the movement type of the note element based on the position of the note element after it moves. Continuing with the above embodiment, the coordinates of the note element in the template quadrant after movement are (-3, -6), (-3, -4), (-5, -6), (-5, -4). The rectangle formed by the vertex coordinates of the moved note element is not inside the template quadrant, and is not inside the note template. At least one processor in the display device determines that the current movement type of the note element is note removal, and then based on the note removal position, the note element can be restored to its original size parameters for display.

[0227] It is understandable that after the removed note element is displayed according to the original size parameters, its original size has a high probability of being larger than the display size parameters within the template quadrant. Therefore, after the size of the note element is enlarged, it is easy to be blocked by the note template. Therefore, it is necessary to adjust the direction of the note element back to the original size parameters according to the moving direction of the note element. That is, at least one processor is specifically configured to execute computer instructions so that the display device displays the note element according to the initial size parameters of the note element by the following execution: marking the direction of the note moving out according to the position transformation information of the note element; setting a scaling point for the note element based on the moving direction of the note; and scaling the note element based on the scaling point and the initial size parameters of the note element.

[0228] As shown in Figure 24, following the above embodiment, at least one processor in the display device can determine that the note element is located in the lower left corner of the template quadrant after moving based on the position transformation information of the note element (the movement amount in the first direction and the movement amount in the second direction). Therefore, the coordinates of the upper right vertex of the note element can be selected as the scaling point, and the note element can be enlarged to the original size parameters and displayed.

[0229] Based on the movement direction of the note element, or the relative positional relationship between the note element and the template quadrant / note template after the movement, the scaling point can be set to avoid occlusion between the enlarged note element and the note template when the note element is enlarged, which is conducive to improving the user experience. It is understandable that the user can drag the note element on the display interface to adjust the position of the note element so that the note element and the note template are no longer obstructed. However, by determining the scaling point before the note element is enlarged to avoid occlusion between the enlarged note element and the note template, the user's operation can be effectively reduced, thereby saving note access time.

[0230] It's understandable that when the moved note element is located in the upper right corner of the template quadrant, the lower left vertex of the note element can be used as the scaling point. If the shape formed by the vertices of the note element is not rectangular, the point closest to the template quadrant can also be used as the scaling point to avoid the enlarged note element and the note template from obscuring each other. As shown in Figure 25, if the note element is circular, the point on the circle closest to the note template can be used as the scaling point, and the circular note template can be restored to its original size parameters based on the scaling point.

[0231] In some embodiments, if at least one processor in a display device determines, based on an operation instruction input by a user, that the movement type of a sticky note element is an inter-quadrant shift, it is necessary to adjust the display format of the sticky note in the template quadrant associated with the movement of the sticky note element. Specifically, if the movement type is an inter-quadrant shift, the size parameters of the sticky note element stored in the template quadrant of the moving note are adjusted based on the number of sticky note elements stored in the template quadrant of the moving note; and the size parameters of the sticky note element stored in the template quadrant of the moving note are adjusted based on the number of sticky note elements stored in the template quadrant of the moving note.

[0232] At least one processor in the display device adjusts the adaptive storage parameters of the template quadrant based on the number of sticky note elements within the template quadrant, and adjusts the display size of the sticky note elements within the template quadrant based on the adaptive storage parameters. As shown in Figure 26, taking template quadrants A and B as examples, sticky note element A is stored in template quadrant A, which stores three sticky note elements, including A; template quadrant B stores two sticky note elements. Based on a user's operation instruction, at least one processor in the display device determines that sticky note element A should be moved from template quadrant A to template quadrant B. At this point, two sticky note elements remain in template quadrant A, and template quadrant B stores three sticky note elements, including sticky note element A.

[0233] At least one processor in the display device adjusts adaptive storage parameters in template quadrants A and B based on changes in the number of sticky note elements in template quadrants A and B. For example, the spacing between sticky note elements in template quadrant A may be increased, while the spacing between sticky note elements in template quadrant B may be decreased. This allows the optimal display state of the sticky note elements in each template quadrant to be set based on the number of sticky note elements.

[0234] It is understood that the quadrant transformation also includes swapping the positions of the sticky note elements in each quadrant, i.e., intra-quadrant transformation of the sticky note elements. When the movement type is intra-quadrant transformation, at least one processor in the display device swaps the positions of the sticky note elements associated with the intra-quadrant transformation within the quadrant.

[0235] As shown in Figure 27, note elements C and D in a template quadrant are used as an example. A user moves note element C to the location of note element D by dragging or other related operations. At least one processor in the display device determines that the current move type of note element C is an intra-quadrant transformation based on the distance before and after the movement, the position after the movement, and the change in the quadrant index. Note elements C and D are then swapped to complete the intra-quadrant transformation.

[0236] S2105: Adjusting the display position of the note element according to the position transformation information, and adjusting the size parameter of the note element according to the display position of the note element.

[0237] As described in the above embodiment, when at least one processor in the display device determines that the movement type of the note element is a note moving in, it can adjust the adaptive storage parameters of the template quadrant according to the number of note elements in the template quadrant, and then adjust the size parameters of the note element and display it. When at least one processor in the display device determines that the movement type of the note element is a note moving out, it can first select a scaling point on the moved note element and adjust the size parameters of the note element. It is understandable that when the note moves out, the processor 250 will adjust the adaptive storage parameters in the template quadrant of the moved note element to adjust the display size of the note element in the template quadrant.

[0238] When at least one processor in the display device determines that the movement type of the sticky note element is a sticky note quadrant transformation, as shown in FIG27 , according to the user's operation instruction, the sticky note element may move to a position intersecting with multiple template quadrants. The processor 250 needs to store the sticky note element in one of the template quadrants. That is, at least one processor is specifically configured to execute computer instructions so that the display device adjusts the display position of the sticky note element according to the position transformation information by performing the following: if the number of template quadrants intersecting with the sticky note element is greater than 1, then mark the movement type as quadrant transformation; calculate the intersection area of ​​the sticky note element and the template quadrant; and store the sticky note element in the template quadrant with the largest intersection area with the sticky note element.

[0239] It should be noted that when the template quadrant intersecting with the note element is 1, at least one processor in the display device can determine that the note element is moved to another template quadrant based on the position transformation information of the note element, so it can also be regarded as performing quadrant transformation on the note element.

[0240] As shown in Figure 28, when a note element intersects with several template quadrants during a quadrant transformation, the area of ​​intersection between the note element and the template quadrant is calculated for each of them. When calculating the area of ​​intersection with the template quadrant, the coordinates of the vertex of the note element and the coordinates of the intersection point between the note element and the template quadrant are obtained and combined with a graphical calculation formula to calculate the intersection area. At least one processor in the display device can sort the calculated intersection areas and select the template quadrant with the largest intersection area to store the note element.

[0241] It can be understood that when the movement type of the note element is note moving in, if it intersects with multiple template quadrants in the note template, at least one processor in the display device can also select the template quadrant with a larger intersection area to store the note element based on the area of ​​intersection between the note element and the template quadrant.

[0242] The intersection area between a note element and a template quadrant can, to some extent, reflect the user's preferred movement when entering an action, specifically the template quadrant in which the note element is most likely to be stored. Therefore, by sorting the intersection area, we can filter out the optimal note storage option to align with the user's intent, thereby reducing the user's effort in adjusting the location of note elements and improving the user experience.

[0243] In some embodiments, the device control method can be applied to the display device 200, the mobile terminal, and the whiteboard software installed in the display device 200 and the mobile terminal. By setting multiple template quadrants of the same type as the note elements in the note template to classify and store the note elements, it is convenient for users to find them. When the note elements are stored in the template quadrant, the display size of the note elements is adjusted according to the number of note elements in the template quadrant to dynamically adjust the display state of the note elements to the optimal state, thereby facilitating user viewing and selection.

[0244] As can be seen from the above technical content, the present disclosure provides a display device and a device control method. The display device receives a note element in response to an operation instruction for creating a new note element. And the note elements are classified and stored in the template quadrant according to the storage identifier of the note element. Within the template quadrant, the size parameters of the note elements in the template quadrant can be adjusted according to the number of note elements. When moving the note element, the display position of the note element can be adjusted according to the position transformation information of the note element, and the size parameters of the note element can be adjusted according to the display position of the note element. The display device classifies and saves the note elements according to the storage identifier to facilitate user search. When the note is moved, the note size parameters in the template quadrant associated with the moved note are adjusted to adjust the note display state to the best, which is convenient for user search.

Claims

1. A display device, comprising: A user input interface configured to receive instructions from a user; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: In response to a window adjustment instruction for a first floating window, determining a first window area according to the window adjustment instruction; presenting at least two floating windows in the display, the first floating window being any one of the two floating windows; Acquire a window display area of ​​a second floating window, where the second floating window is a floating window in the user interface other than the first floating window; If the first window area intersects with the window display area, calculating a second window area of ​​the second floating window, where the second window area is used to represent a window display area that does not intersect with the first window area, and the second window area is calculated based on the positional relationship between the first window area and the window display area; The display is controlled to display the first floating window in the first window area, and to display the second floating window in the second window area.

2. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a start instruction of a new floating window, a floating window list is obtained through a database interface; Reading the window coordinates of the displayed floating window in the floating window list, and calculating the displayed area according to the window coordinates, wherein the displayed area is the area where the displayed floating window is displayed; Removing the displayed area from the screen display area of ​​the display to obtain a non-displayed area; Locating a target display area in the non-display area based on a floating window area, the floating window area being an application display area of ​​the new floating window; The new floating window is created according to the target display area.

3. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Dividing the screen display area into a plurality of unit grids; Marking the unit grids outside the displayed area with a first identification grid, and marking the unit grids within the displayed area with a second identification grid; The target display area is located in the non-display area based on the floating window area, and the target display area is a display area when all unit grids covered by the floating window area are the first identification grids.

4. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Obtaining the center point coordinates of the target display area and the center point coordinates of the screen display area; Calculating the center distance between the center point coordinates of the target display area and the center point coordinates of the screen display area; An adapted display area is determined according to the center distance, and the new floating window is created in the adapted display area; the adapted display area is a target display area whose center distance is less than a preset center threshold.

5. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: When the target display area is not located, scaling the floating window area according to a preset ratio; The target display area is located in the non-display area based on the scaled floating window area.

6. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Detecting a window operation of the window adjustment instruction, wherein the window operation includes a move operation, a zoom-in operation, and a zoom-out operation; If the window operation is a moving operation or a zooming operation, executing a step of determining a first window area according to the window adjustment instruction; If the window operation is a zoom-out operation, the first floating window is adjusted to the first window area.

7. The display device according to claim 6, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Detecting the starting coordinates and real-time coordinates in the window adjustment instruction; Obtaining the center coordinates of the first floating window; Calculating a first distance between the center coordinate and the starting coordinate, and calculating a second distance between the center coordinate and the real-time coordinate; If the first distance is smaller than the second distance, marking the window operation of the window adjustment instruction as a zoom-in operation; If the first distance is greater than the second distance, the window operation of the window adjustment instruction is marked as a zoom-out operation.

8. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Acquire the reference coordinates of the first floating window, the center coordinates of the first window area, and the center coordinates of the second floating window; Calculating a first relative distance between the reference coordinates of the first floating window and the center coordinates of the second floating window, and calculating a second relative distance between the center coordinates of the first window area and the center coordinates of the second floating window; If the first relative distance is less than or equal to the second relative distance, adjusting the first floating window to the first window area; If the first relative distance is greater than the second relative distance, a step of calculating a second window area of ​​the second floating window is performed.

9. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Calculating a positional relationship between the first window area and the window display area, wherein the positional relationship includes intersection and non-intersection; Moving the window display area to obtain a second window area when the position relationship is non-intersecting; When the second window area is not obtained, reducing the window display area according to a preset reduction ratio; A second window area when the positional relationship is non-intersecting is acquired based on the reduced window display area.

10. The display device according to any one of claims 1 to 9, wherein the at least one processor is further configured to execute computer instructions to enable the display device to execute: In response to an operation instruction for creating a new note element, receiving a note element; storing the note element in a template quadrant of a note template according to the storage identifier of the note element; According to the number of sticky note elements in the template quadrant, adjusting the size parameters of the sticky note elements in the template quadrant; In response to an operation instruction for moving a sticky note element, acquiring position change information of the sticky note element; The display position of the note element is adjusted according to the position transformation information, and the size parameter of the note element is adjusted according to the display position of the note element.

11. The display device according to claim 10, wherein the at least one processor is specifically configured to execute computer instructions so that the display device stores the note element into a template quadrant of the note template according to the storage identifier of the note element by performing the following steps: If the number of the sticky note elements stored in the template quadrant is greater than the storage number threshold, the sticky note elements are stored in the non-template quadrant of the sticky note template; If the number of note elements stored in the template quadrant is less than or equal to the storage quantity threshold, The storage identifier of the element stores the note element in the template quadrant of the note template.

12. The display device according to claim 11, wherein the at least one processor is specifically configured to execute computer instructions so that the display device adjusts the size parameters of the sticky note elements in the template quadrant according to the number of sticky note elements stored in the template quadrant by performing the following: Calculating adaptive storage parameters of the template quadrant according to the number of note elements stored in the template quadrant, the adaptive storage parameters including note row spacing, note column spacing, and edge spacing; Based on the adaptive storage parameters, adjusting the size parameters of the sticky note elements in the template quadrant; The display is controlled to display the sticky note element in the template quadrant.

13. The display device according to claim 10, wherein the at least one processor is specifically configured to execute computer instructions so that the display device receives a note element in response to an operation instruction for creating a new note element by executing: If it is detected that the display interface of the display includes a sticky note template, a step of storing the sticky note element in a template quadrant of the sticky note template according to the storage identifier of the sticky note element is performed; If it is detected that the display interface of the display does not include a note template, the display is controlled to display the received note element.

14. The display device according to claim 10, wherein the at least one processor is specifically configured to execute computer instructions so that the display device obtains the position transformation information of the sticky note element in response to the operation instruction for moving the sticky note element by executing the following: If the display interface of the display includes a sticky note element and a sticky note template, then parsing the movement type of the move operation, the movement type is used to characterize the relative movement mode between the sticky note element and the sticky note template, and the movement type includes sticky note moving in, sticky note moving out, sticky note quadrant transformation, and sticky note quadrant transformation; If the movement type is a sticky note move-in, storing the sticky note element in a template quadrant that intersects with the sticky note element; If the movement type is note removal, displaying the note element according to the initial size parameter of the note element; If the movement type is inter-quadrant transformation, then according to the number of note elements stored in the template quadrant of the moved-in note, the size parameters of the note elements stored in the template quadrant of the moved-in note are adjusted; and according to the number of note elements stored in the template quadrant of the moved-out note, the size parameters of the note elements stored in the template quadrant of the moved-out note are adjusted; If the movement type is an intra-quadrant transformation, the positions of the sticky note elements associated with the intra-quadrant transformation are swapped within the quadrant.

15. The display device according to claim 14, wherein the at least one processor is specifically configured to execute computer instructions so that the display device obtains the position transformation information of the sticky note element in response to the operation instruction for moving the sticky note element by executing the following: Before performing a move operation on the note element, reading the quadrant index of the note element; If the note element is not stored in the note template, obtaining position change information of the note element when the move operation is performed, the position change information including the first direction movement amount and the second direction movement amount of the note element; If the note element intersects with a template quadrant in the note template, the movement type is marked as a note move-in.

16. The display device according to claim 14, wherein the at least one processor is specifically configured to execute computer instructions so that the display device adjusts the display position of the note element according to the position transformation information by performing the following: Before performing a move operation on the note element, reading the quadrant index of the note element; If the sticky note element is located in the template quadrant, obtaining position change information of the sticky note element when performing a move operation; If the note element does not intersect with the template quadrant in the note template, the movement type is marked as note removal.

17. The display device according to claim 16, wherein the at least one processor is specifically configured to execute computer instructions so that the display device displays the note element according to the initial size parameter of the note element by performing the following: According to the position change information of the sticky note element, mark the direction in which the sticky note moves out; Based on the direction in which the sticky note is moved out, setting a scaling point for the sticky note element; The note element is scaled based on the scaling point and an initial size parameter of the note element.

18. The display device according to claim 16, wherein the at least one processor is specifically configured to execute computer instructions so that the display device adjusts the display position of the note element according to the position transformation information by performing the following: If the number of template quadrants intersecting the note element is greater than 1, marking the movement type as quadrant transformation; Calculating the intersection area between the note element and the template quadrant; The note element is stored in the template quadrant that has the largest intersection area with the note element.

19. A device control method, comprising: In response to a window adjustment instruction for the first floating window, determining a first window area according to the window adjustment instruction; At least two floating windows are presented in the display, and the first floating window is any one of the two floating windows; Acquire a window display area of ​​a second floating window, where the second floating window is a floating window other than the first floating window in the user interface of the display; If the first window area intersects with the window display area, calculating a second window area of ​​the second floating window, where the second window area is used to represent a window display area that does not intersect with the first window area, and the second window area is calculated based on the positional relationship between the first window area and the window display area; The display is controlled to display the first floating window in the first window area, and to display the second floating window in the second window area.

Images