Screen content-based multi-screen display and interaction method, device and terminal

Abstract

The invention provides a screen content-based multi-screen display and interaction method and device and a terminal, and the method can respond to a display instruction to obtain multiple groups of display data, and carries out the matching with the preset data of a screen according to the display demands of each group of display data. Therefore, the display tasks of the main screen, the auxiliary screen and the screens which display different contents are determined, the to-be-displayed contents are re-determined based on the current display contents and the interaction instruction after the main screen triggers the interaction instruction, and then new display data are generated and the display requirements of the screens are re-matched. According to the scheme provided by the invention, the display tasks are reasonably allocated, so that a plurality of screens can be more effectively utilized, the interactivity of multi-screen display and the flexibility of information display are effectively improved, the correlation and the visual effect of display contents are improved, more reasonable display content management, good user experience and efficient resource utilization are realized, and the user experience is improved. And the working efficiency and the information circulation speed can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-screen display, and particularly relates to a multi-screen display and interaction method and device based on screen content and a terminal. BACKGROUND

[0002] With the rapid development of information technology, display technology is also constantly innovating, especially in intelligent terminals and multimedia devices. As the core component of information display, the screen is no longer limited to a single display mode. Multi-screen display technology, with its ability to display different content simultaneously, has gradually become an important development direction for modern intelligent terminals, computers, televisions and other devices, and has been widely used in fields such as collaborative office, entertainment experience and smart home.

[0003] However, existing multi-screen display technology still faces some technical challenges. Most traditional multi-screen display solutions are limited to copying the same content to multiple screens or simply switching the display content between multiple screens. In dealing with complex information and interaction, it is often difficult to fully utilize the display potential of each screen, making it difficult to achieve accurate matching and flexible scheduling of content.

[0004] In addition, current multi-screen display technology often ignores intelligent allocation and dynamic adjustment of display tasks, resulting in a lack of flexibility and high relevance in display content between screens, making it difficult to make timely feedback and adjustments according to user needs or operations. For example, in a collaborative office or multitasking scenario, users may need to display different data and interfaces on multiple screens at the same time, but existing technology lacks intelligent processing in screen content allocation and adjustment, often relying on preset configurations and failing to meet complex interaction needs. SUMMARY

[0005] To overcome the above-mentioned deficiencies of the prior art, the present application proposes a multi-screen display and interaction method based on screen content, applied to a terminal comprising at least two screens, the method comprising: In response to a display instruction, a plurality of sets of display data containing to-be-displayed content are obtained, and the display requirements of each set of display data are determined to match each set of display data with the preset data of each screen; Based on the matching result, the display screen for displaying the to-be-displayed content is determined, at least one of the display screens is determined as the main screen for displaying core content and / or interaction interface, and the other display screens are determined as the auxiliary screens for displaying extended content and / or auxiliary information, so that each display screen displays the to-be-displayed content contained in the matched display data; In response to the interaction instruction triggered in the interaction interface of any main screen, the content to be displayed is re-determined based on the interaction instruction, the main screen information applied by the interaction instruction, and the current display content of each display screen related to the interaction instruction, and a plurality of sets of display data are generated based on the currently determined content to be displayed, so as to re-execute the step of "determining the display requirements of each set of display data, and matching each set of display data with the preset data of each screen".

[0006] Preferably, the method further comprises: If there is a plurality of display data matching a plurality of display screens, the plurality of associated screen groups are determined based on the display screens matching the same set of display data. Based on the preset data and the preset user preference settings of each associated screen group, the part of the content to be displayed applied to each display screen in the associated screen group corresponding to the matching set of display data is determined and displayed accordingly.

[0007] Further, the method further comprises: If the content to be displayed contained in the same set of display data matching a plurality of display screens includes a cross-screen continuous picture, the arrangement position data and the pixel density data of each display screen matching the display data are obtained. Based on the arrangement position data and the pixel density data of each display screen matching the display data, the display data is interpolated to enable the cross-screen continuous picture to be displayed continuously on each display screen matching the display data.

[0008] Specifically, the "step of re-determining the content to be displayed based on the interaction instruction, the main screen information applied by the interaction instruction, and the current display content of each display screen related to the interaction instruction in response to the interaction instruction triggered in the interaction interface of any main screen, and generating a plurality of sets of display data based on the currently determined content to be displayed, so as to re-execute the step of "determining the display requirements of each set of display data, and matching each set of display data with the preset data of each screen", comprises: A virtual reality logic layer is established for each associated screen group containing a main screen and a sub-screen, to record the display attributes and content association relationship of each screen in the group. In response to the interaction instruction including the cross-screen instruction of switching part of the display content of the main screen to the display of the secondary screen, the data priority of the cross-screen content and the non-cross-screen content in the main screen is compared; when the cross-screen content has higher data priority, the data of the non-cross-screen content in the main screen is removed from the display task of the main screen and added to the display task of the secondary screen, the data of all display content in the secondary screen is removed from the display task of the secondary screen and added to the display task of the main screen, the previous main screen is re-determined as the secondary screen, the previous secondary screen is re-determined as the main screen, and the corresponding display attribute and content association relationship in the virtual reality logical layer is updated; If the non-cross-screen content has higher data priority, the data of the cross-screen content in the main screen is removed from the display task of the main screen and added to the display task of the secondary screen, and the corresponding display attribute and content association relationship in the virtual reality logical layer is updated; In response to the interaction instruction including the cross-screen instruction of switching part of the display content of the secondary screen to the display of the main screen and / or fusing part of the display content of the secondary screen with the display content of the main screen, the data of the cross-screen content in the secondary screen is removed from the display task of the secondary screen and added to the display task of the main screen, and the corresponding display attribute and content association relationship in the virtual reality logical layer is updated.

[0009] Specifically, the "determining the display requirements of each group of display data based on the preset rules to match each group of display data with the preset data of each screen" comprises: measuring the visual redundancy of each group of display data by the formula calculating the visual redundancy threshold of each pixel in each group of display data to determine the tolerable error range of each group of display data in visual quality; wherein, the visual redundancy threshold of each pixel is represented by V, the luminance level is represented by L, the contrast threshold is represented by C, the blur masking threshold is represented by B, the directionality is represented by D, and is calculated by the following formula respectively:

[0010]

[0011]

[0012]

[0013] wherein, the position of the image is represented by P, brightness value, Represents the contrast ratio. Indicates the image at position The gradient change value, This indicates the location of the blurred image. brightness value, Represents a set of highly sensitive pixels. Represents the set of pixels with medium sensitivity. Represents a set of low-sensitivity pixels. The weights representing the set of highly sensitive pixels. The weights representing the set of medium-sensitivity pixels, The weights representing the set of low-sensitivity pixels; Based on the tolerable error range of visual quality for each set of display data, the visual perception quality of each set of display data on different screens is evaluated, and the optimal display parameters for each set of display data on different screens are determined so as to match each set of display data with the preset data of each screen.

[0014] Preferably, the interaction commands include one or more of touch commands, gesture commands, voice commands, mouse commands, keyboard commands, and eye-tracking commands; the method further includes: In response to an interactive command, the display state of each sub-screen associated with the interactive command is recorded, so that after the response step to the interactive command is completed, the display state of each sub-screen associated with the interactive command changes after the interactive command occurs. If there is a secondary screen whose display state has changed, obtain the change data of the secondary screen whose display state has changed, and analyze whether the change in the display state of the secondary screen is related to the interaction command based on the change data; Sub-screens whose display state changes are associated with the interaction command are identified as associated sub-screens. The relationship between the command type of the interaction command and the display state change data of the associated sub-screens is recorded. Based on the relationship, the display adjustment strategy of the sub-screens associated with the interaction command with the same command type in the relationship is optimized.

[0015] Preferably, the interaction commands include one or more of touch commands, gesture commands, voice commands, mouse commands, keyboard commands, and eye-tracking commands; the method further includes: In response to an interactive command, the display state of each sub-screen associated with the interactive command is recorded, so that after the response step to the interactive command is completed, the display state of each sub-screen associated with the interactive command changes after the interactive command occurs. If there is a secondary screen whose display state has changed, obtain the change data of the secondary screen whose display state has changed, and analyze whether the change in the display state of the secondary screen is related to the interaction command based on the change data; Sub-screens whose display state changes are associated with the interaction command are identified as associated sub-screens. The relationship between the command type of the interaction command and the display state change data of the associated sub-screens is recorded. Based on the relationship, the display adjustment strategy of the sub-screens associated with the interaction command with the same command type in the relationship is optimized.

[0016] Furthermore, the phrase "determining at least one of the display screens as the main screen for displaying core content and / or the interactive interface, while determining the other display screens as secondary screens for displaying extended content and / or auxiliary information" includes: The display screen that matches the display data that is assigned to the first priority is determined as the main screen; The display screen that matches the display data that has been assigned to the second priority is identified as the secondary screen; Based on the data content of the display data that is classified as the third priority, the display screen that matches the display data that is classified as the third priority is determined as the main screen or the secondary screen.

[0017] The present invention also proposes a multi-screen display and interaction device based on screen content, applicable to a terminal including at least two screens, the device comprising: The matching module is used to respond to display commands, obtain multiple sets of display data containing the content to be displayed, determine the display requirements of each set of display data, and match each set of display data with the preset data of each screen; The display module is used to determine the display screen for displaying the content to be displayed based on the matching result, and to determine at least one of the display screens as the main screen for displaying core content and / or interactive interface, while determining other display screens as secondary screens for displaying extended content and / or auxiliary information, so that each of the display screens displays the content to be displayed contained in the matching display data accordingly; An interaction module is used to respond to an interaction command triggered on any main screen's interactive interface, and to redetermine the content to be displayed based on the interaction command, the main screen information applied by the interaction command, and the current display content of each of the display screens related to the interaction command. It also generates multiple sets of display data based on the currently determined content to be displayed, so that the matching module re-executes the step of "determining the display requirements of each set of display data to match each set of display data with the preset data of each screen".

[0018] The present invention also proposes a terminal comprising at least two screens for implementing the multi-screen display and interaction method based on screen content as described above.

[0019] The present invention has at least the following beneficial effects: The solution proposed in this invention rationally distributes different types of content across multiple screens, enabling users to obtain more intuitive and richer information displays on multiple display screens. This improves users' understanding of the content and interactive experience, reduces information overload, avoids excessive concentration and clutter of information, and allows complex data and content to be presented more clearly and structurally. This helps improve the efficiency of information management and data analysis, and can adjust the displayed content on each screen in real time in response to changes in interactive commands. This allows users to switch smoothly between different screens according to their actual needs, and makes the information across multiple screens more coordinated and consistent, improving the naturalness of interaction and response speed. Furthermore, the solution proposed in this invention can determine multiple associated screen groups based on the same group of display data, and can perform interpolation processing on continuous cross-screen images by combining the arrangement position and pixel density data of the display screens, effectively eliminating display gaps between different screens, achieving smooth transition of images, thereby further optimizing the display effect, improving the visual consistency between screens, and allowing users to experience a more natural display effect when operating multiple screens, ensuring the reasonable allocation and coordination of display content; the method can also re-evaluate and adjust the content to be displayed, combining multi-screen collaboration with user interaction needs, optimizing the allocation of display tasks and content updates, improving the intelligent response capability of the system, and improving the user's operating efficiency and experience; Furthermore, the proposed solution evaluates display data based on visual redundancy measurement. It can determine the allowable error range when displaying on different screens according to the visual redundancy threshold of the displayed content, and optimize the matching method of display data, thereby reducing visual errors and information loss. It can also intelligently detect changes in the display status of the secondary screen and optimize the response strategy for changes in display status by recording the relationship between interactive commands and the display status of the secondary screen. The dynamic allocation method of screen tasks according to the priority of the display content can effectively utilize the display capacity of each screen, avoid the crowding or chaos of the display content, improve the utilization efficiency of screen resources, make the display content more reasonable, and avoid the omission or unclear display content.

[0020] Therefore, the present invention provides a method, device and terminal for multi-screen display and interaction based on screen content. The solution proposed by the present invention can make more effective use of multiple screens by reasonably allocating display tasks, effectively improve the interactivity of multi-screen display and the flexibility of information display, improve the relevance and visual effect of displayed content, achieve more reasonable display content management, good user experience and efficient resource utilization, and help improve work efficiency and information flow speed. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a multi-screen display and interaction method based on screen content, as provided in Embodiment 1. Figure 2 A flowchart illustrating the method for prioritizing and matching displayed data; Figure 3 A flowchart illustrating a method for recording interactive commands and associating them with the secondary screen display status to optimize display adjustment strategies; Figure 4 This is a schematic diagram of the module structure of a multi-screen display and interaction device based on screen content, as provided in Embodiment 2.

[0023] Figure Labels 10-Matching module; 11-Division unit; 12-Matching unit; 20-Display module; 30-Association module; 40-Interaction module; 50-Recording module; 60-Analysis module; 70-Optimization module. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] Various embodiments of the invention will be described more fully below. The invention may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the invention to the specific embodiments disclosed herein, but rather the invention should be understood to cover all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the invention.

[0026] In the following, the terms “comprising” or “may include” as used in various embodiments of the invention indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of the invention, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0027] In various embodiments of the invention, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0028] The expressions used in the various embodiments of the present invention (such as "first," "second," etc.) may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used for the purpose of distinguishing one element from other elements. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first element may be referred to as a second element without departing from the scope of the various embodiments of the present invention, and similarly, a second element may also be referred to as a first element.

[0029] It should be noted that, in this invention, unless otherwise explicitly specified and defined, terms such as "installation," "connection," and "fixation" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] In this invention, those skilled in the art should understand that the terms indicating orientation or positional relationship in the text are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the purpose of facilitating the description of this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0031] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0032] Example 1 Please see Figures 1-3 This embodiment proposes a multi-screen display and interaction method based on screen content, applicable to terminals including at least two screens. The method includes: S100: In response to a display command, acquire multiple sets of display data containing the content to be displayed, determine the display requirements of each set of display data, and match each set of display data with the preset data of each screen.

[0033] In this embodiment, the preset data for each screen may include, but is not limited to, physical location data. It should be noted that the human eye has limited ability to distinguish blur; a blur effect needs to reach a certain degree before it can be distinguished by the human eye. Based on this, in this embodiment, step S100 measures the visual redundancy of each group of display data. Specifically, this is achieved by calculating the visual redundancy threshold of each pixel in each group of display data to determine the tolerable error range of visual quality for each group of display data. The formula for calculating the visual redundancy threshold is as follows:

[0034] in, This represents the visual redundancy threshold for each pixel. Indicates brightness level, Indicates the contrast threshold. Indicates the fuzzy masking threshold. Indicates directionality. The results are obtained using the following formulas:

[0035]

[0036]

[0037]

[0038] in, Indicates the image at position brightness value, Represents the contrast ratio. Indicates the image at position The gradient change value, This indicates the location of the blurred image. brightness value, Represents a set of highly sensitive pixels. Represents the set of pixels with medium sensitivity. Represents a set of low-sensitivity pixels. The weights representing the set of highly sensitive pixels. The weights representing the set of medium-sensitivity pixels, The weights represent the set of low-sensitivity pixels.

[0039] By dividing pixels into high-sensitivity, medium-sensitivity, and low-sensitivity pixel sets and assigning different weights to each set, the visual sensitivity differences in different regions of an image can be processed more accurately, ensuring the display quality of key areas while appropriately optimizing low-sensitivity areas to reduce redundant calculations and resource waste.

[0040] Therefore, step S100 can evaluate the visual perception quality of each set of display data on different screens based on the tolerable error range of each set of display data in visual quality, determine the optimal display parameters of each set of display data on different screens, and match each set of display data with the preset data of each screen. By calculating the visual redundancy threshold of each pixel in each set of display data, the method proposed in this embodiment fully considers the characteristics of the human visual system. It can dynamically adjust the matching strategy of display data according to factors such as image brightness, contrast, blur masking, and directionality, so as to better simulate human visual perception of images. This achieves a certain degree of tolerable error in visual quality, so as to optimize the overall display effect and system performance through matching without affecting user perception.

[0041] S200: Based on the matching result, determine the display screens for displaying the content to be displayed, and determine at least one of the display screens as the main screen for displaying core content and / or interactive interface, while determining the other display screens as secondary screens for displaying extended content and / or auxiliary information, so that each display screen displays the content to be displayed contained in the matching display data accordingly.

[0042] If there is matching group display data for multiple display screens, proceed with steps S300 and S400 sequentially; if there is no matching group display data for multiple display screens, proceed directly with step S400.

[0043] S300: Based on the display screens that match the same group of display data, determine multiple associated screen groups. Based on the preset data and preset user preference settings of each associated screen group, determine the part of the content to be displayed that is applied to each display screen in the associated screen group corresponding to the same group of display data, and display it accordingly.

[0044] Specifically, if the content to be displayed in the same group of display data matching multiple display screens in step S300 includes cross-screen continuous images, the method proposed in this embodiment will obtain the arrangement position data and pixel density data of each display screen that matches the display data, and perform interpolation processing on the display data based on the arrangement position data and pixel density data of each display screen that matches the display data, so that the cross-screen continuous images are displayed continuously on each display screen that matches the display data.

[0045] S400: In response to an interaction command triggered on any main screen's interactive interface, the content to be displayed is re-determined based on the interaction command, the main screen information of the interaction command application, and the current display content of each display screen related to the interaction command. Multiple sets of display data are generated based on the currently determined content to be displayed, so as to re-execute the step of "determining the display requirements of each set of display data and matching each set of display data with the preset data of each screen" in step S100.

[0046] Specifically, step S100, "determining the display requirements of each group of display data to match each group of display data with the preset data of each screen," includes: S110: Obtain the display requirements of each group of display data, and divide each group of display data into one of the following priorities from high to low: first priority, second priority, and / or third priority, based on the display requirements of each group of display data, and the amount of display data divided into first priority is not greater than the preset amount of data.

[0047] S120: Based on the display data that is classified as first priority, the display data that is classified as second priority, and the display data that is classified as third priority, the preset data of each screen are matched in sequence.

[0048] Accordingly, the step S200, which involves "determining at least one of the display screens as the main screen for displaying core content and / or the interactive interface, while determining the other display screens as secondary screens for displaying extended content and / or auxiliary information," includes: The display screen that matches the display data that is classified as first priority is determined as the main screen; The display screen that matches the display data that has been classified as second priority is identified as the secondary screen; Based on the data content of the display data that is classified as the third priority, the display screen that matches the display data that is classified as the third priority is determined as the main screen or the secondary screen.

[0049] By assigning different priorities to the displayed data, it is possible to achieve orderly processing of the displayed data and screens, and avoid some screens wasting resources by displaying low-priority content. Based on the matching mechanism of data priority and display screen, different needs can be flexibly responded to, the load of each display screen can be better controlled, the accuracy of information display and priority management can be improved, thus bringing a smoother operating experience and stability. In different scenarios, it is also possible to adjust which display screen displays which content as needed. Meanwhile, setting the amount of data to be displayed with the highest priority not to exceed the preset amount of data ensures that the display of high-priority information will not exceed the capacity of the display screen, thereby avoiding screen congestion or information distortion caused by too much information. This allows users to clearly obtain important information and enhances the user experience.

[0050] Specifically, step S400 includes: A virtual reality logic layer is established for each group of associated screens that simultaneously contains a main screen and a secondary screen. This layer is used to record the display attributes and content relationships of each screen within the group. In response to interactive commands including switching part of the content displayed on the main screen to the secondary screen, compare the data priority of cross-screen content and non-cross-screen content within the main screen; If cross-screen content has higher data priority, remove the data of non-cross-screen content in the main screen from the main screen's display task and add it to the secondary screen's display task. Remove the data of all displayed content in the secondary screen from the secondary screen's display task and add it to the main screen's display task. This will re-determine the existing main screen as the secondary screen and re-determine the existing secondary screen as the main screen. At the same time, update the corresponding display attributes and content associations in the virtual reality logic layer. If non-cross-screen content has a higher data priority, the data of cross-screen content in the main screen will be removed from the main screen's display task and added to the secondary screen's display task. At the same time, the corresponding display attributes and content associations in the virtual reality logic layer will be updated. In response to interactive commands including switching part of the content displayed on the secondary screen to the main screen and / or merging part of the content displayed on the secondary screen with the content displayed on the main screen, the data of the cross-screen content in the secondary screen is removed from the secondary screen's display task and added to the main screen's display task, while updating the corresponding display attributes and content associations in the virtual reality logic layer.

[0051] Therefore, the method proposed in this embodiment manages the display attributes and content associations of each screen through the virtual reality logic layer, ensuring that the display tasks of each screen can be flexibly adjusted according to needs. This enables the system to intelligently switch screen tasks and schedule content, enhancing the system's adaptability. By comparing the data priorities of the content in the main screen and the secondary screen, it determines which content to display first, ensuring that more important data always occupies the main screen, while less important data can be transferred to the secondary screen for display, improving the efficiency and accuracy of information display. By intelligently managing the display tasks of the main and secondary screens, the load is balanced across different screens. Regardless of how the content on the screen changes, the virtual reality logic layer will update in a timely manner to ensure the consistency and accuracy of the display status, avoid content errors or incorrect display, improve the level of intelligent interaction, and provide richer interaction methods in complex environments such as multi-screen interaction and virtual reality.

[0052] Preferably, the interaction commands may include, but are not limited to, touch commands, gesture commands, voice commands, mouse commands, keyboard commands, and eye-tracking commands. The method proposed in this embodiment also includes: S500: In response to an interactive command, it records the display state of each sub-screen related to the interactive command, so that after the response steps to the interactive command are completed, it can detect whether the display state of each sub-screen related to the interactive command has changed after the interactive command occurred.

[0053] S600: If there is a secondary screen whose display state has changed, obtain the change data of the secondary screen whose display state has changed, and analyze whether the change in the display state of the secondary screen is related to the interaction command based on the change data.

[0054] S700: The secondary screens that are associated with the display state change and the interaction command are identified as associated secondary screens. The relationship between the command type of the interaction command and the display state change data of the associated secondary screen is recorded. Based on the relationship, the display adjustment strategy of the secondary screens associated with the interaction command with the same command type in the relationship is optimized.

[0055] Through steps S500-S700, the method proposed in this embodiment can record and analyze the display status changes of the secondary screen, dynamically respond to interactive commands, enable the system to adjust the display strategy in a timely manner, ensure that the user's interactive behavior can be responded to and fed back quickly, help to gradually accumulate user interactive behavior and display feedback, continuously improve the display effect and system response, enhance the intelligence level of the terminal, and provide a more personalized and efficient user experience.

[0056] Example 2 Please see Figure 4This embodiment proposes a multi-screen display and interaction device based on screen content, used to implement the multi-screen display and interaction method based on screen content proposed in Embodiment 1. The device includes: The matching module 10 is used to respond to the display command, acquire multiple sets of display data containing the content to be displayed, determine the display requirements of each set of display data, and match each set of display data with the preset data of each screen; Display module 20 is used to determine the display screen for displaying the content to be displayed based on the matching result, and to determine at least one of the display screens as the main screen for displaying the core content and / or interactive interface, while determining the other display screens as the secondary screens for displaying extended content and / or auxiliary information, so that each display screen displays the content to be displayed contained in the matching display data accordingly. The association module 30 is used to determine multiple associated screen groups based on the display screens that match the same group of display data, and to determine the part of the content to be displayed for each display screen in the associated screen group corresponding to the matching same group of display data based on the preset data and preset user preference settings of each associated screen group, and to display it accordingly. The interaction module 40 is used to respond to an interaction command triggered on any main screen's interactive interface, redetermine the content to be displayed based on the interaction command, the main screen information of the interaction command application, and the current display content of each display screen related to the interaction command, and generate multiple sets of display data based on the currently determined content to be displayed, so that the matching module re-executes the step of "determining the display requirements of each set of display data, so as to match each set of display data with the preset data of each screen".

[0057] In this embodiment, the matching module 10 performs visual redundancy measurement on each group of display data. Specifically, it calculates the visual redundancy threshold of each pixel in each group of display data to determine the tolerable error range of visual quality for each group of display data. Based on the tolerable error range of visual quality for each group of display data, it evaluates the visual perception quality of each group of display data on different screens and determines the optimal display parameters for each group of display data on different screens, so as to match each group of display data with the preset data of each screen. The formula for calculating the visual redundancy threshold is:

[0058] in, This represents the visual redundancy threshold for each pixel. Indicates brightness level, Indicates the contrast threshold. Indicates the fuzzy masking threshold. Indicates directionality. The results are obtained using the following formulas:

[0059]

[0060]

[0061]

[0062] in, Indicates the image at position brightness value, Represents the contrast ratio. Indicates the image at position The gradient change value, This indicates the location of the blurred image. brightness value, Represents a set of highly sensitive pixels. Represents the set of pixels with medium sensitivity. Represents a set of low-sensitivity pixels. The weights representing the set of highly sensitive pixels. The weights representing the set of medium-sensitivity pixels, The weights represent the set of low-sensitivity pixels.

[0063] Specifically, if the content to be displayed in the same group of display data matching multiple display screens includes a continuous cross-screen image, the association module 30 will obtain the arrangement position data and pixel density data of each display screen that matches the display data, and perform interpolation processing on the display data based on the arrangement position data and pixel density data of each display screen that matches the display data, so that the continuous cross-screen image is displayed continuously on each display screen that matches the display data.

[0064] Specifically, the matching module 10 includes: The division unit 11 is used to obtain the display requirements of each group of display data, and to divide each group of display data into one of the first priority, second priority and / or third priority with high priority to low based on the display requirements of each group of display data, and the amount of display data divided into the first priority is not greater than the preset amount of data. The matching unit 12 is used to match the display data divided into first priority, second priority, and third priority with the preset data of each screen in sequence. Accordingly, the display module 20 will determine the display screen that matches the display data that has been assigned the first priority as the main screen; The display screen that matches the display data that has been classified as second priority is identified as the secondary screen; Based on the data content of the display data that is classified as the third priority, the display screen that matches the display data that is classified as the third priority is determined as the main screen or the secondary screen.

[0065] Specifically, the interaction module 40 will establish a virtual reality logic layer for each group of associated screens that simultaneously contains a main screen and a secondary screen, for recording the display attributes and content relationships of each screen in the group, thereby responding to interaction commands including cross-screen commands that switch part of the display content of the main screen to the secondary screen, and comparing the data priority of cross-screen content and non-cross-screen content in the main screen. If cross-screen content has higher data priority, remove the data of non-cross-screen content in the main screen from the main screen's display task and add it to the secondary screen's display task. Remove the data of all displayed content in the secondary screen from the secondary screen's display task and add it to the main screen's display task. This will re-determine the existing main screen as the secondary screen and re-determine the existing secondary screen as the main screen. At the same time, update the corresponding display attributes and content associations in the virtual reality logic layer. If non-cross-screen content has a higher data priority, the data of cross-screen content in the main screen will be removed from the main screen's display task and added to the secondary screen's display task. At the same time, the corresponding display attributes and content associations in the virtual reality logic layer will be updated. Alternatively, in response to an interaction command including switching part of the content displayed on the secondary screen to the main screen and / or merging part of the content displayed on the secondary screen with the content displayed on the main screen, the data of the cross-screen content in the secondary screen is removed from the display task of the secondary screen and added to the display task of the main screen, while updating the corresponding display attributes and content associations in the virtual reality logic layer.

[0066] Preferably, the interaction commands may include, but are not limited to, touch commands, gesture commands, voice commands, mouse commands, keyboard commands, and eye-tracking commands. The device proposed in this embodiment also includes: Recording module 50: In response to an interactive command, it records the display state of each sub-screen related to the interactive command, so as to detect whether the display state of each sub-screen related to the interactive command has changed after the interactive command is executed after the response steps to the interactive command are completed.

[0067] The analysis module 60 is used to obtain the change data of the secondary screen whose display state has changed when there is a secondary screen whose display state has changed, so as to analyze whether the change in the display state of the secondary screen is related to the interaction command based on the change data.

[0068] The optimization module 70 is used to identify the secondary screens whose display state changes are related to the interaction commands as associated secondary screens, record the association between the command type of the interaction command and the display state change data of the associated secondary screens, and optimize the display adjustment strategy of the secondary screens associated with the interaction commands that have the same command type in the association relationship based on the association relationship.

[0069] Example 3 This embodiment proposes a terminal for implementing the multi-screen display and interaction method based on screen content proposed in Embodiment 1.

[0070] In summary, this invention provides a multi-screen display and interaction method, device, and terminal based on screen content. The proposed solution can more effectively utilize multiple screens by rationally allocating display tasks, effectively improving the interactivity and information display flexibility of multi-screen displays, enhancing the relevance and visual effects of displayed content, achieving more reasonable display content management, a better user experience, and efficient resource utilization, which helps to improve work efficiency and information flow speed.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-screen display and interaction method based on screen content, characterized in that, Applied to a terminal comprising at least two screens, the method includes: In response to a display command, multiple sets of display data containing the content to be displayed are obtained, the display requirements of each set of display data are determined, and each set of display data is matched with the preset data of each screen. Based on the matching results, a display screen is determined for displaying the content to be displayed. At least one of the display screens is determined as the main screen for displaying core content and / or interactive interface, while the other display screens are determined as secondary screens for displaying extended content and / or auxiliary information, so that each of the display screens displays the content to be displayed contained in the matching display data accordingly. In response to an interaction command triggered on any main screen's interactive interface, the content to be displayed is re-determined based on the interaction command, the main screen information applied by the interaction command, and the current display content of each of the display screens related to the interaction command. Multiple sets of display data are generated based on the currently determined content to be displayed, so as to re-execute the step of "determining the display requirements of each set of display data to match each set of display data with the preset data of each screen".

2. The method according to claim 1, characterized in that, The method further includes: If there is display data in the same group that matches multiple display screens, determine multiple associated screen groups based on the display screens that match the display data in the same group; Based on the preset data and preset user preference settings of each associated screen group, the portion of the content to be displayed in each display screen of the associated screen group that matches the display data of the same group is determined and displayed accordingly.

3. The method according to claim 2, characterized in that, The method further includes: If the content to be displayed in the same group of display data matching multiple display screens includes continuous images across screens, obtain the arrangement position data and pixel density data of each display screen that matches the display data; Based on the arrangement position data and pixel density data of each display screen that match the display data, the display data is interpolated to ensure that the cross-screen continuous image is displayed coherently on each display screen that matches the display data.

4. The method according to claim 2 or 3, characterized in that, The step of "responding to an interaction command triggered on any main screen's interactive interface, re-determining the content to be displayed based on the interaction command, the main screen information applied by the interaction command, and the current display content of each of the display screens related to the interaction command, and generating multiple sets of display data based on the currently determined content to be displayed, so as to re-execute the step of "determining the display requirements of each set of display data, so as to match each set of display data with the preset data of each screen" includes: A virtual reality logic layer is established for each group of associated screens that simultaneously contains a main screen and a secondary screen. This layer is used to record the display attributes and content relationships of each screen within the group. In response to the interaction instruction including a cross-screen instruction to switch part of the display content of the main screen to the secondary screen, the data priorities of the cross-screen content and the non-cross-screen content in the main screen are compared; when the cross-screen content has a higher data priority, the data of the non-cross-screen content in the main screen is removed from the display task of the main screen and added to the display task of the secondary screen, and the data of all display content in the secondary screen is removed from the display task of the secondary screen and added to the display task of the main screen, so that the primary screen is re-determined as the secondary screen and the secondary screen is re-determined as the primary screen, while updating the corresponding display attributes and content associations in the virtual reality logic layer; If the non-cross-screen content has a higher data priority, then the data of the cross-screen content in the main screen is removed from the display task of the main screen and added to the display task of the secondary screen, while updating the corresponding display attributes and content associations in the virtual reality logic layer. In response to the interaction instruction including a cross-screen instruction that switches part of the content displayed on the secondary screen to the main screen and / or merges part of the content displayed on the secondary screen with the content displayed on the main screen, the data of the cross-screen content in the secondary screen is removed from the display task of the secondary screen and added to the display task of the main screen, while updating the corresponding display attributes and content associations in the virtual reality logic layer.

5. The method according to claim 1, characterized in that, The phrase "determining the display requirements of each group of display data based on preset rules, so as to match each group of display data with the preset data of each screen" includes: Visual redundancy was measured for each set of displayed data using a formula. Calculate the visual redundancy threshold for each pixel in each set of display data to determine the tolerable error range in visual quality for each set of display data; in, This represents the visual redundancy threshold for each pixel. Indicates brightness level, Indicates the contrast threshold. Indicates the fuzzy masking threshold. Directionality is indicated by the following formulas: in, Indicates the image at position brightness value, Represents the contrast ratio. Indicates the image at position The gradient change value, This indicates the location of the blurred image. brightness value, Represents a set of highly sensitive pixels. Represents the set of pixels with medium sensitivity. Represents a set of low-sensitivity pixels. The weights representing the set of highly sensitive pixels. The weights representing the set of medium-sensitivity pixels, The weights representing the set of low-sensitivity pixels; Based on the tolerable error range of visual quality for each set of display data, the visual perception quality of each set of display data on different screens is evaluated, and the optimal display parameters for each set of display data on different screens are determined so as to match each set of display data with the preset data of each screen.

6. The method according to claim 1, characterized in that, The interactive commands include one or more of touch commands, gesture commands, voice commands, mouse commands, keyboard commands, and eye-tracking commands; the method further includes: In response to an interactive command, the display state of each sub-screen associated with the interactive command is recorded, so that after the response step to the interactive command is completed, the display state of each sub-screen associated with the interactive command changes after the interactive command occurs. If there is a secondary screen whose display state has changed, obtain the change data of the secondary screen whose display state has changed, and analyze whether the change in the display state of the secondary screen is related to the interaction command based on the change data; Sub-screens whose display state changes are associated with the interaction command are identified as associated sub-screens. The relationship between the command type of the interaction command and the display state change data of the associated sub-screens is recorded. Based on the relationship, the display adjustment strategy of the sub-screens associated with the interaction command with the same command type in the relationship is optimized.

7. The method according to claim 1 or 5, characterized in that, The phrase "determining the display requirements of each group of display data, so as to match each group of display data with the preset data of each screen" includes: The display requirements of each group of display data are obtained, and each group of display data is divided into one of the following priorities from high to low: first priority, second priority, and / or third priority, based on the display requirements of each group of display data. The amount of display data divided into the first priority is not greater than the preset amount of data. The display data assigned to the first priority, the display data assigned to the second priority, and the display data assigned to the third priority are sequentially matched with the preset data of each screen.

8. The method according to claim 7, characterized in that, The phrase "designating at least one of the display screens as the main screen for displaying core content and / or the interactive interface, while designating other display screens as secondary screens for displaying extended content and / or auxiliary information" includes: The display screen that matches the display data that is assigned to the first priority is determined as the main screen; The display screen that matches the display data that has been assigned to the second priority is identified as the secondary screen; Based on the data content of the display data that is classified as the third priority, the display screen that matches the display data that is classified as the third priority is determined as the main screen or the secondary screen.

9. A multi-screen display and interaction device based on screen content, characterized in that, The device, applicable to a terminal comprising at least two screens, includes: The matching module is used to respond to display commands, obtain multiple sets of display data containing the content to be displayed, determine the display requirements of each set of display data, and match each set of display data with the preset data of each screen; The display module is used to determine the display screen for displaying the content to be displayed based on the matching result, and to determine at least one of the display screens as the main screen for displaying core content and / or interactive interface, while determining other display screens as secondary screens for displaying extended content and / or auxiliary information, so that each of the display screens displays the content to be displayed contained in the matching display data accordingly; The interaction module is used to respond to an interaction command triggered on any main screen's interactive interface, and to redetermine the content to be displayed based on the interaction command, the main screen information applied by the interaction command, and the current display content of each of the display screens related to the interaction command. It also generates multiple sets of display data based on the currently determined content to be displayed, so that the matching module re-executes the step of "determining the display requirements of each set of display data to match each set of display data with the preset data of each screen".

10. A terminal, characterized in that, It includes at least two screens for implementing the method as described in any one of claims 1-8.

Images