Screen-flying interaction method of multi-screen interaction system and multi-screen interaction system

Abstract

The invention relates to a screen flying interaction method of a multi-screen interaction system and the multi-screen interaction system. The method comprises the steps that when it is detected that a target gesture is adopted to operate a screen flying element on a source screen, a screen flying process is started, the screen flying element is any interactive object displayed on the source screen, and the screen flying process is that the element displayed on the source screen is displayed on a target screen or preset operation is executed; determining a target screen, and sending the associated data of the screen flying element to the target screen; and displaying a screen flying element or executing a preset operation on the target screen based on the associated data. The technical problems that an existing multi-screen display system lacks a convenient and visual cross-screen interaction means, and a user is difficult to directly operate elements on one screen and generate corresponding effects on another screen are solved.

Description

Technical Field

[0001] The present application relates to the fields of multimedia presentation and human-computer interaction, and in particular to a flying screen interaction method and a multi-screen interactive system. Background Art

[0002] With the development of information technology, multi-screen display systems are increasingly being adopted to improve information delivery efficiency and user experience. However, most current multi-screen display systems lack convenient and intuitive cross-screen interaction methods. In particular, touch interfaces make it difficult for users to directly manipulate elements on one screen and produce corresponding effects on another. Therefore, it is necessary to develop a novel flying screen interaction mechanism to improve the user interaction experience and enhance the practicality of multi-screen display systems. Summary of the Invention

[0003] The flying screen interaction method and multi-screen interactive system provided in the embodiments of the present application at least solve the problem that the existing multi-screen display system lacks convenient and intuitive cross-screen interaction means, and it is difficult for users to directly operate elements on one screen and produce corresponding effects on another screen.

[0004] According to one aspect of an embodiment of the present application, a flying screen interaction method of a multi-screen interactive system is provided, the multi-screen interactive system including multiple display screens and at least one host, the multiple display screens being connected to the host, the method including: when it is detected that a target gesture is used to operate a flying screen element on a source screen, starting a flying screen process, wherein the flying screen element is any interactive object displayed on the source screen, and the flying screen process is to display the element displayed on the source screen on the target screen or perform a preset operation; determine the target screen, and send associated data of the flying screen element to the target screen; and display the flying screen element on the target screen or perform a preset operation based on the associated data.

[0005] Optionally, determining the target screen includes at least one of the following: pre-setting a fixed screen as the target screen; setting the screen selected by the user from the first window displayed on the source screen as the target screen, wherein the first window is used to display all available screens; determining the target screen based on a target gesture; selecting the screen closest to the flying screen element as the target screen; determining the target screen based on the user's usage habits and historical records; determining the target screen based on the user's preferences in different time periods based on a machine learning algorithm; determining the target screen based on the correlation between the content of the flying screen element and the content displayed on each display screen; determining the target screen based on the type of task or application to which the flying screen element belongs.

[0006] Optionally, before starting the flying screen process, the above method also includes: displaying a second window on the source screen, wherein the second window is used to prompt the user to confirm whether to start the flying screen process; if it is detected that the user confirms to start the flying screen process, the flying screen process is started; if it is detected that the user cancels the start of the flying screen process, the flying screen process is refused to start.

[0007] Optionally, before sending the associated data of the flying screen element to the target screen, the method further includes: serializing the flying screen element and its associated data and converting them into a target data format; and adding a timestamp or an identifier of the source screen to the data in the target data format.

[0008] Optionally, the associated data of the flying screen element is sent to the target screen, including: if multiple display screens are connected to a host, the source screen sends the associated data of the flying screen element to the target screen through the UDP protocol or the graphics card driver; if each of the multiple display screens is connected to a host, the host connected to the source screen uses the UDP protocol or the TCP protocol to send the associated data of the flying screen element to the host connected to the target screen, and the host connected to the target screen sends the associated data of the flying screen element to the target screen.

[0009] Optionally, displaying a flying screen element or performing a preset operation on the target screen based on the associated data includes: after verifying the associated data, parsing the associated data according to preset rules; and rebuilding the user interface component corresponding to the flying screen element on the target screen according to the parsing result.

[0010] Optionally, displaying the flying screen element or executing a preset operation on the target screen based on the associated data further includes: playing an animation on the target screen, wherein the animation is used to simulate a process of sending the flying screen element from the source screen to the target screen.

[0011] Optionally, if each of the multiple display screens is connected to a host, before starting the flying screen process, the above method also includes: after each host is started, broadcasting its own device information through the network, and listening to the broadcast messages sent by other hosts, and establishing a device list based on the monitored broadcast messages; each host is paired and authenticated with the hosts in its device list.

[0012] According to another aspect of an embodiment of the present application, a multi-screen interactive system is also provided, including: multiple display screens and a host, wherein the multiple display screens are connected to the host and are configured to display flying screen elements, and the flying screen element is any interactive object displayed on the multiple display screens; the host is configured to start a flying screen process when it detects that a target gesture is used to operate a flying screen element on a source screen among the multiple display screens, wherein the flying screen process is to display the elements displayed on the source screen on the target screen or perform a preset operation; determine the target screen from the multiple display screens, and send the associated data of the flying screen element to the target screen; display the flying screen element on the target screen or perform a preset operation based on the associated data.

[0013] According to another aspect of an embodiment of the present application, a multi-screen interactive system is also provided, including: multiple display screens and multiple hosts, wherein each of the multiple display screens is connected to a host respectively, and the multiple display screens are configured to display flying screen elements, and the flying screen elements are any interactive objects displayed on the multiple display screens; a first host is connected to a source screen among the multiple display screens, and is configured to start a flying screen process when it detects that a target gesture is used to operate the flying screen elements on the source screen, wherein the flying screen process is to display the elements displayed on the source screen on the target screen or perform a preset operation; determine the target screen from the multiple display screens, and send the associated data of the flying screen elements to the host connected to the target screen; a second host is connected to the target screen, and is configured to send the associated data of the flying screen elements to the target screen, and display the flying screen elements on the target screen or perform a preset operation based on the associated data.

[0014] According to another aspect of the embodiments of the present application, a non-transitory machine-readable medium storing computer instructions is further provided, where the computer instructions are used to enable a computer to execute the above method.

[0015] Beneficial effects of the embodiments of the present application:

[0016] In an embodiment of the present application, a flying screen interaction method of a multi-screen interactive system is provided. The multi-screen interactive system includes multiple display screens and at least one host, and the multiple display screens are connected to the host. The method includes: when it is detected that a target gesture is used to operate a flying screen element on a source screen, starting a flying screen process, wherein the flying screen element is any interactive object displayed on the source screen, and the flying screen process is to display the element displayed on the source screen on the target screen or perform a preset operation; determine the target screen, and send the associated data of the flying screen element to the target screen; display the flying screen element on the target screen or perform a preset operation based on the associated data, send the element on one screen to another screen through a simple gesture operation, and trigger the corresponding link page display or other action, thereby improving the interactivity and fun of the multi-screen display system; simplifying the cross-screen operation process and reducing the user's cognitive burden; enhancing the coherence and consistency of the displayed content and improving the overall visual impact; and expanding the application scenario range of the multi-screen display system. Technical effect.

[0017] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other embodiments can be derived from these drawings without inventive effort.

[0019] Figure 1 This is a flow chart of a flying screen interaction method of a multi-screen interactive system according to an embodiment of the present application;

[0020] Figure 2 is a structural diagram of a multi-screen interactive system according to an embodiment of the present application;

[0021] Figure 3 is a structural diagram of another multi-screen interactive system according to an embodiment of the present application;

[0022] Figure 4 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0023] The following describes embodiments of the present invention in more detail with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] Source screen: refers to the initial screen that initiates the flying screen operation.

[0026] Target screen: refers to the screen that receives elements from the source screen and responds to its instructions.

[0027] Flying screen elements: can be any interactive objects, such as buttons and pictures.

[0028] In the related art, most multi-screen display systems lack convenient and intuitive cross-screen interaction methods. Especially in touch interfaces, it is difficult for users to directly operate elements on one screen and produce corresponding effects on another screen. To solve this problem, the embodiments of this application provide relevant solutions, which are described in detail below.

[0029] Figure 1 This is a flow chart of a flying screen interaction method of a multi-screen interactive system according to an embodiment of the present application. The multi-screen interactive system includes multiple display screens and at least one host. The multiple display screens are connected to the host, such as Figure 1 As shown, the method includes the following steps:

[0030] Step S102, when it is detected that a target gesture is used to operate a flying screen element on the source screen, the flying screen process is started, wherein the flying screen element is any interactive object displayed on the source screen, and the flying screen process is to display the element displayed on the source screen on the target screen or perform a preset operation.

[0031] In an embodiment of the present application, a plurality of elements with a fly screen capability are determined on a source screen, and when a user performs a specific gesture operation on a selected element, the system captures the action and starts a fly screen process. The source screen is a touch screen.

[0032] On the source screen, you can detect user touch behavior through event monitoring mechanisms (such as touchstart, touchmove, and touchend) in JavaScript or other programming languages. Analyze the user's gesture pattern to determine whether it meets the preset trigger conditions for the fly-through process. For example, if the user's gesture is a quick movement in a specific direction of the screen or a long press and swipe, it meets the preset trigger conditions for the fly-through process.

[0033] Step S104: determine the target screen, and send the associated data of the flying screen element to the target screen.

[0034] In this step, the target screen can be determined by pre-configuration or dynamic detection, and a data packet containing the flying screen element and its associated data information (including but not limited to position, size, style and carried data) is sent to the target screen through an appropriate communication protocol (such as UDP, HDMI extension, etc.).

[0035] Step S106: displaying a flying screen element or executing a preset operation on the target screen based on the associated data.

[0036] The target screen parses the received data packet and presents the corresponding link page or performs other specified operations in a set manner.

[0037] The above method provided in the embodiment of the present application sends elements on one screen to another screen through simple gesture operations, and triggers the display of corresponding link pages or other actions, thereby improving the interactivity and fun of the multi-screen display system; simplifying the cross-screen operation process and reducing the user's cognitive burden; enhancing the coherence and consistency of the displayed content and improving the overall visual impact; and expanding the scope of application scenarios of the multi-screen display system.

[0038] According to some optional embodiments of the present application, executing step S104 to determine the target screen can be achieved by the following method:

[0039] 1) Pre-set a fixed screen as the target screen.

[0040] A fixed fly screen target screen is pre-configured in the system or application. Each time a fly screen operation is performed, the system directly sends data to the preset screen.

[0041] This method is suitable for scenarios with a fixed multi-screen layout and a clear target screen. For example, in a conference room, the main screen is used to display the main content, and the secondary screen is used to display notes or extended information. The advantage is that it is simple to implement and does not require dynamic detection.

[0042] 2) The screen selected by the user from the first window displayed on the source screen is used as the target screen, wherein the first window is used to display all selectable screens.

[0043] After the fly screen operation is triggered, a small window pops up on the source screen, listing all available screens for the user to choose from. The user confirms the fly screen target by clicking or selecting the target screen.

[0044] This method is suitable for scenarios where users need to explicitly specify the target screen, such as in an exhibition hall where users need to send content to a specific display screen. The advantage is that users have strong autonomy and can avoid misoperation.

[0045] 3) Determine the target screen based on the target gesture.

[0046] The user specifies the target screen through a specific gesture (such as drag direction). For example, dragging to the left means sending to the left screen, and dragging upwards means sending to the upper screen.

[0047] This method is suitable for scenarios where the screen layout is relatively regular and users are familiar with gesture operations. The advantage is that the operation is intuitive and reduces the number of user selection steps.

[0048] 4) Select the screen closest to the flying screen element as the target screen.

[0049] The system automatically selects the closest screen as the target screen based on the initial position of the fly screen element and the screen layout. For example, the system determines the closest screen by calculating the distance between the center points of the screens.

[0050] This method is suitable for scenarios where the screen layout is dense and users want to quickly complete the flying screen operation. The advantage is that it is highly automated and reduces user operations.

[0051] 5) Determine the target screen based on the user's usage habits and history.

[0052] The system automatically recommends the most likely target screen based on the user's usage habits and history. For example, if the user frequently sends content to a certain screen, the system will prioritize that screen as the target.

[0053] This method is suitable for scenarios that require a personalized experience, such as personal work environments or multi-screen systems that are used for a long time. The advantage is that it can improve the user experience and reduce repetitive operations.

[0054] 6) Determine the target screen based on user preferences at different time periods based on machine learning algorithms.

[0055] The system uses machine learning algorithms to analyze the user's flying screen operation patterns and dynamically adjust the target screen selection logic. For example, it automatically adjusts the target screen based on the user's preferences at different time periods.

[0056] This approach is suitable for scenarios that require a high degree of intelligence, such as smart conference room systems. Its advantage is its high flexibility and ability to adapt to changes in user habits.

[0057] 7) Determine the target screen based on the relevance between the content of the flying screen elements and the display content of each display screen.

[0058] The system selects the target screen based on the relevance of the fly-screen element's content to the target screen's display content. For example, if the fly-screen element is a document, the system automatically sets the target screen to the screen currently displaying the document's related application.

[0059] This approach is suitable for scenarios requiring intelligent content distribution based on context, such as multi-screen office environments. Its advantages include a high degree of intelligence and improved content presentation consistency.

[0060] 8) Determine the target screen based on the type of task or application to which the flying screen element belongs.

[0061] Select the target screen based on the task or application type to which the fly-on screen element belongs. For example, automatically send video content to the screen displaying a media player.

[0062] This method is suitable for multi-tasking environments, such as creative studios or control centers. Its advantage is that it can improve task management efficiency.

[0063] The above describes several methods for determining the target screen.

[0064] As an optional embodiment of the present application, before executing step S102 to start the flying screen process, a second window can also be displayed on the source screen, wherein the second window is used to prompt the user to confirm whether to start the flying screen process; if it is detected that the user confirms to start the flying screen process, the flying screen process is started; if it is detected that the user cancels the start of the flying screen process, the flying screen process is refused to start.

[0065] In this embodiment, a pop-up window mode is supported. Before executing the fly-through process, a small window pops up on the source screen to ask the user whether to continue. After receiving a positive response, the request is formally initiated. The pop-up window can contain a brief description, a cancel button, and a confirmation button. The design should be concise and clear to avoid interfering with the main task flow.

[0066] In addition, in the embodiments of the present application, flexible configuration options are provided to allow users to customize different types of flying screen behaviors, such as single click, long press trigger, etc.; and it has good compatibility and scalability, and is suitable for screen combinations of various sizes and resolutions.

[0067] In other optional embodiments of the present application, before executing step S104 to send the associated data of the flying screen element to the target screen, the flying screen element and its associated data need to be serialized and converted into the target data format; and a timestamp or an identifier of the source screen is added to the data in the target data format.

[0068] Once the fly-screen action is determined, the elements involved and their associated data (such as ID, location, size, style, and URL) need to be serialized and converted into a data format suitable for network transmission (JSON, XML, Protobuf, etc.). JSON is suitable for structured data and is easy to parse. XML is suitable for complex data structures. Protobuf is suitable for high-performance, low-latency scenarios.

[0069] In some optional embodiments, some additional information needs to be added, such as a timestamp, a source screen identifier, etc., to facilitate subsequent processing.

[0070] According to some optional embodiments of the present application, step S104 is executed to send the associated data of the flying screen element to the target screen, which is achieved by the following method: if multiple display screens are connected to a host, the source screen sends the associated data of the flying screen element to the target screen via the UDP protocol or the graphics card driver.

[0071] For a system with multiple screens connected to one host (i.e., multiple screens connected to one host), if the multiple screens are connected via a local area network, the User Datagram Protocol (UDP) can be used to send data because it has a lower latency characteristic; if a more complex instruction set is involved, the Transmission Control Protocol (TCP) can be used to send data.

[0072] If multiple displays are connected using a High Definition Multimedia Interface (HDMI) extension, the frame buffer can be shared directly using the API provided by the graphics card driver, thereby achieving seamless switching.

[0073] Additionally, use WebSocket or other persistent connection technologies to ensure real-time, two-way communication, especially when frequent status updates need to be exchanged. WebSocket is a protocol that enables full-duplex communication over a single TCP connection. It simplifies data exchange between clients and servers, allowing servers to proactively push data to clients.

[0074] If each of the multiple display screens is connected to a host, the host connected to the source screen uses the UDP protocol or TCP protocol to send the associated data of the flying screen elements to the host connected to the target screen, and the host connected to the target screen sends the associated data of the flying screen elements to the target screen.

[0075] For multi-machine and multi-screen scenarios (each screen is equipped with an independent host, and elements are delivered between the screens of multiple hosts), the implementation method of multi-screen operation transmission between multiple hosts is more complicated, involving communication between different hosts, data synchronization, and cross-device interaction logic.

[0076] Similarly, if multiple hosts are connected via a local area network, the User Datagram Protocol (UDP) can be used to send data because it has lower latency characteristics; if a more complex instruction set is involved, the Transmission Control Protocol (TCP) can be used to send data.

[0077] Additionally, use WebSockets or other persistent connection technologies to ensure real-time two-way communication, especially if frequent state updates need to be exchanged.

[0078] If multiple hosts are connected across a network, data transmission can be performed via RESTful APIs or GraphQL. Alternatively, a lightweight message queue protocol, Message Queuing Telemetry Transport (MQTT), can be used for data transmission.

[0079] According to some optional embodiments of the present application, step S106 is executed to display a flying screen element or perform a preset operation on the target screen based on the associated data, including the following steps: after verifying the associated data, parsing the associated data according to preset rules; and reconstructing the user interface component corresponding to the flying screen element on the target screen according to the parsing result.

[0080] The target screen's application listens to a designated port, waiting for data packets from the source screen. Upon receiving the data, it verifies its integrity and legitimacy, then parses the content according to established rules and rebuilds the corresponding UI components.

[0081] As an optional embodiment of the present application, when the received data contains link information, a built-in function provided by the browser, such as window.location.href, can be used to load a new page. This method is simple and direct and is suitable for situations where a complete jump to another URL is required.

[0082] For some specific application scenarios, dedicated APIs may be called to load pages. These APIs may provide more control options, such as cache management and request methods, to enable more precise control over page loading behavior.

[0083] For internal page jumps, this can be accomplished by modifying the DOM structure or activating hidden page fragments to ensure a smooth and natural transition.

[0084] Modifying the DOM structure means dynamically adding, deleting, or updating page elements through JavaScript, which can update partial content without refreshing the entire page. This is very effective for improving user experience because it reduces loading time and avoids page flickering issues.

[0085] Activating hidden fragments means that different "subpages" or blocks of content of a page are preloaded and hidden. When the user performs a certain action (such as clicking a menu item), the corresponding hidden parts are activated and displayed, while other parts may be hidden. This approach is often used in single-page applications, where most of the content is loaded asynchronously via AJAX or similar technologies, and the main body of the page is not reloaded.

[0086] AJAX is a technology used to create fast, dynamic web pages. By exchanging data with the server and updating parts of the web page without reloading the entire page, AJAX makes the web experience smoother and more efficient.

[0087] As an optional embodiment of the present application, step S106 is executed to display the flying screen element or perform a preset operation on the target screen based on the associated data, and an animation can also be played on the target screen, where the animation is used to simulate the process of sending the flying screen element from the source screen to the target screen.

[0088] In some optional embodiments of the present application, an animation effect can also be played on the target screen to simulate the process of the flying screen element "flying" from the source screen to the current position of the target screen, thereby achieving the effect of enhancing visual feedback.

[0089] In the embodiments of this application, the UI design of the source screen should include intuitive and easy-to-understand operation prompts to guide users on how to activate the fly screen function. The fly screen elements should have obvious visual characteristics to facilitate distinction and selection, and support multiple state changes (normal, pressed state, flying state).

[0090] The target screen UI design should ensure that when new elements arrive, there is enough space to display them without obscuring important information. If it is a full-screen replacement, appropriate transition animations should be considered, such as fade-in and fade-out, zooming, etc., to improve the overall experience.

[0091] In an embodiment of the present application, after the fly-screen process is completed, the user can continue to interact with the fly-screen elements on the target screen. The state of the elements (such as clicks, scrolling, etc.) is synchronized between the source screen and the target screen in real time via the network. The target screen feeds back the user's operation to the source screen to ensure that the status of both screens is consistent.

[0092] In another optional embodiment of the present application, if each of the multiple display screens is connected to a host, the following operations need to be performed before executing step S102 to start the flying screen process: after each host is started, its respective device information is broadcast through the network, and broadcast messages sent by other hosts are listened to, and a device list is established based on the listened broadcast messages; each host is paired and authenticated with the hosts in its device list.

[0093] In a multi-machine, multi-screen environment, the first thing to do is to enable each host to discover each other. This can be achieved through broadcast / multicast protocols (such as Simple Service Discovery Protocol, SSDP) or service registration and discovery mechanisms (such as mDNS, Multicast DNS).

[0094] When each host starts up, it broadcasts its device information (such as IP address, device name, supported functions, etc.) over the network. Other hosts listen to these broadcast messages and build a device list. For example, in a home or office environment, using the mDNS protocol allows devices to automatically discover each other within the local area network.

[0095] To ensure security, hosts need to be paired and authenticated. This can be done using PIN code pairing, QR code scanning, or authentication based on public key encryption. The process is as follows:

[0096] 1) The user initiates a pairing request through a host.

[0097] 2) The target device displays pairing information (such as a PIN code or QR code).

[0098] 3) The user enters pairing information on the initiating host to complete the binding between the hosts.

[0099] It is similar to the pairing process of Bluetooth devices or AirPlay.

[0100] The flying screen interaction method of the multi-screen interactive system provided in this application can be applied to the following scenarios:

[0101] Meeting scenario: Wirelessly project the contents of a laptop onto a large screen in a meeting room.

[0102] Educational scenario: Teachers can control multiple screens in the classroom using a tablet computer from the podium.

[0103] Exhibition scenario: Visitors can send content to the display screen via handheld devices.

[0104] Figure 2 is a structural diagram of a multi-screen interactive system according to an embodiment of the present application, such as Figure 2 As shown, the system includes: multiple display screens 20 and a host 22, wherein,

[0105] A plurality of display screens 20 are connected to a host 22 and are configured to display flying screen elements. A flying screen element is any interactive object displayed on the plurality of display screens.

[0106] The host 22 is configured to start a flying screen process when it detects that a target gesture is used to operate a flying screen element on a source screen among multiple display screens, wherein the flying screen process is to display the elements displayed on the source screen on the target screen or perform a preset operation; determine the target screen from the multiple display screens, and send the associated data of the flying screen element to the target screen; and display the flying screen element on the target screen or perform a preset operation based on the associated data.

[0107] Figure 2 The multi-screen interactive system shown is the one-machine multi-screen system mentioned above. Figure 2 The preferred implementation of the embodiment shown can be referred to Figure 1 Description of the illustrated embodiment.

[0108] Figure 3 is a structural diagram of another multi-screen interactive system according to an embodiment of the present application, such as Figure 3 As shown, the system includes: a plurality of display screens 30 and a plurality of hosts 32, wherein each of the plurality of display screens 30 is connected to a host 32.

[0109] The multiple display screens 30 are configured to display flying screen elements, where a flying screen element is any interactive object displayed on the multiple display screens.

[0110] A first host is connected to a source screen among the plurality of display screens 30 and is configured to initiate a flying screen process when detecting an operation on a flying screen element on the source screen using a target gesture, wherein the flying screen process is to display the element displayed on the source screen on the target screen or perform a preset operation; determine the target screen from the plurality of display screens, and send associated data of the flying screen element to the host connected to the target screen;

[0111] The second host is connected to the target screen and is configured to send associated data of the flying screen element to the target screen, and display the flying screen element on the target screen or perform a preset operation based on the associated data.

[0112] Figure 3The multi-screen interactive system shown is the multi-machine multi-screen system mentioned above. Figure 3 The preferred implementation of the embodiment shown can be referred to Figure 1 Description of the illustrated embodiment.

[0113] An embodiment of the present application further provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform the method of the embodiment of the present application.

[0114] An embodiment of the present application further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0115] An embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0116] refer to Figure 4 , a structural block diagram of an electronic device that can be used as a server or client of an embodiment of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0117] like Figure 4 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the electronic device can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0118] Multiple components within the electronic device are connected to the I / O interface 405, including an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device capable of inputting information into the electronic device. The input unit 406 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device capable of presenting information and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include, but is not limited to, a magnetic disk or an optical disk. The communication unit 409 allows the electronic device to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0119] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units for running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0120] The computer programs for implementing the methods of the embodiments of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of embodiments of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic or infrared systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the foregoing.

[0122] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0123] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0124] The various steps described in the method implementation schemes provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation schemes may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.

[0125] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments are referenced to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the partial description of the method embodiment.

[0126] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A flying screen interaction method for a multi-screen interactive system, characterized in that: The multi-screen interactive system includes a plurality of display screens and at least one host, wherein the plurality of display screens are connected to the host, and the method includes: When a target gesture is detected to be used to operate a flying screen element on the source screen, a flying screen process is started, wherein the flying screen element is any interactive object displayed on the source screen, and the flying screen process is to display the element displayed on the source screen on the target screen or perform a preset operation; Determine the target screen, and send the associated data of the flying screen element to the target screen; The flying screen element is displayed on the target screen or the preset operation is performed based on the associated data.

2. The method according to claim 1, characterized in that Determining the target screen includes at least one of the following: Presetting a fixed screen as the target screen; taking a screen selected by a user from a first window displayed on the source screen as the target screen, wherein the first window is used to display all selectable screens; determining the target screen based on the target gesture; Selecting the screen closest to the flying screen element as the target screen; Determine the target screen according to the user's usage habits and history records; Determining the target screen based on user preferences at different time periods based on a machine learning algorithm; Determining the target screen according to the relevance between the content of the flying screen element and the display content of each display screen; The target screen is determined according to the type of task or application to which the flying screen element belongs.

3. The method according to claim 1, characterized in that Before starting the flying screen process, the method further includes: Displaying a second window on the source screen, wherein the second window is used to prompt the user to confirm whether to start the flying screen process; If it is detected that the user confirms to start the flying screen process, the flying screen process is started; If it is detected that the user cancels the start of the flying screen process, the start of the flying screen process is rejected.

4. The method according to claim 1, wherein Before sending the associated data of the flying screen element to the target screen, the method further includes: Serializing the flying screen elements and their associated data into a target data format; A timestamp or an identifier of the source screen is added to the data in the target data format.

5. The method according to claim 1, wherein Sending the associated data of the flying screen element to the target screen includes: If the multiple display screens are connected to one host, the source screen sends the associated data of the flying screen element to the target screen via the UDP protocol or the graphics card driver; If each of the multiple display screens is connected to a host, the host connected to the source screen uses the UDP protocol or the TCP protocol to send the associated data of the flying screen element to the host connected to the target screen, and the host connected to the target screen sends the associated data of the flying screen element to the target screen.

6. The method according to claim 1, characterized in that Displaying the flying screen element or executing the preset operation on the target screen based on the associated data includes: After verifying the associated data, parsing the associated data according to preset rules; The user interface component corresponding to the flying screen element is rebuilt on the target screen according to the parsing result.

7. The method according to claim 1 or 6, characterized in that Displaying the flying screen element or executing the preset operation on the target screen based on the associated data further includes: Animation is played on the target screen, wherein the animation is used to simulate a process of sending the flying screen element from the source screen to the target screen.

8. The method according to claim 1, characterized in that If each of the plurality of display screens is connected to a host, before starting the flying screen process, the method further includes: After each host is started, it broadcasts its own device information through the network, listens to broadcast messages sent by other hosts, and establishes a device list based on the listened broadcast messages; Each host is paired and authenticated with the hosts in its device list.

9. A multi-screen interactive system, characterized in that: include: Multiple displays and a host, where The multiple display screens are connected to the host and configured to display flying screen elements, where the flying screen elements are any interactive objects displayed on the multiple display screens; The host is configured to start a flying screen process when it detects that a target gesture is used to operate a flying screen element on a source screen among the multiple display screens, wherein the flying screen process is to display the elements displayed on the source screen on the target screen or perform a preset operation; determine the target screen from the multiple display screens and send the associated data of the flying screen element to the target screen; and display the flying screen element on the target screen or perform the preset operation based on the associated data.

10. A multi-screen interactive system, characterized in that: include: Multiple display screens and multiple hosts, wherein each of the multiple display screens is connected to a host, The multiple display screens are configured to display a flying screen element, where the flying screen element is any interactive object displayed on the multiple display screens; A first host is connected to a source screen among the multiple display screens and is configured to initiate a flying screen process when detecting an operation on a flying screen element on the source screen using a target gesture, wherein the flying screen process is to display the element displayed on the source screen on a target screen or perform a preset operation; determine the target screen from the multiple display screens, and send associated data of the flying screen element to the host connected to the target screen; The second host is connected to the target screen and is configured to send the associated data of the flying screen element to the target screen, and display the flying screen element or perform the preset operation on the target screen based on the associated data.

11. A non-transitory machine-readable medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

Images