Multi-screen display method, WEB terminal device, storage medium and program product

Abstract

The present invention discloses a multi-screen display method, a WEB-end device, a storage medium, and a program product, which relate to the field of data display technology, including: a main screen control module on the WEB end cuts image picture information representing updates on multiple screens into main screen picture information and sub-screen picture information, adjusts the size of the main screen picture information according to the size change of the main screen browser viewport area, and displays the size-adjusted main screen picture information in the changed main screen browser viewport area; a sub-screen processing module on the WEB end adjusts the size of the sub-screen picture information according to the size change of the sub-screen browser viewport area, and displays the size-adjusted sub-screen picture information in the changed sub-screen browser viewport area. In this way, the method of the present invention does not require the installation of special client software and only requires modification on the WEB end. It can be widely applied to multi-tasking and remote collaboration scenarios, meets users' needs for efficient display methods, and greatly optimizes resource utilization.

Description

Technical Field

[0001] The present disclosure generally relates to data display technology, and more particularly to a multi-screen display method, a WEB terminal device, a storage medium, and a program product. Background Art

[0002] The current method for realizing multi-screen display of cloud desktop pages can be to install special client software in the electronic device and access it through the client software; or, the virtual machine can be accessed through the World Wide Web (WEB) browser. In this case, the server usually merges the multi-screen image information and transmits it to the WEB end, and finally the WEB end compresses and displays all multi-screen virtual images in a single browser viewport area; the former multi-screen display method is not only cumbersome in the display process, but also prone to driver compatibility issues when applied across operating systems. The latter multi-screen display method has the problem that the screen display content will overflow as the browser viewport area changes, resulting in greater limitations of the existing multi-screen display method. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the existing technology, it is hoped to provide a multi-screen display method, WEB-side device, storage medium and program product. The WEB-side only needs to be modified with the help of the native support of the browser and does not need to install special client software. It can run on all platforms and realize the multi-screen split-screen display effect of the virtual machine through the browser / server (B / S architecture), which solves the limitations of the existing technology in multi-screen display on the WEB side. Moreover, the size of the updated screen information on different screens can also be adaptively adjusted as the corresponding screen browser viewport changes, ensuring that different screen information can be displayed completely and clearly under different browser viewport sizes, avoiding problems such as screen stretching and deformation, reducing server pressure and network resource consumption, and realizing accurate screen presentation of multi-screen display. It can be widely used in multi-tasking and remote collaboration scenarios, meeting users' needs for efficient display methods, significantly improving user experience, and greatly optimizing resource utilization.

[0004] In a first aspect, the present invention provides a multi-screen display method, which is applied to a WEB terminal, wherein the WEB terminal includes a main screen control module and at least one secondary screen processing module, and the main screen control module interacts with the secondary screen processing module and the server respectively for data; the method includes:

[0005] The main screen control module, when determining that the image information sent by the server indicates that multiple screens have been updated, divides the image information into main screen information and sub-screen information based on the screen configuration information; and, when monitoring a change in the main screen browser viewport area, adjusts the size of the main screen information based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area;

[0006] When the secondary screen processing module detects a change in the secondary screen browser viewport area, the secondary screen processing module adjusts the size of the secondary screen image information based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area;

[0007] The main screen control module controls the main screen image information after size adjustment to be displayed in the current main screen browser viewport area;

[0008] The secondary screen processing module displays the secondary screen image information after size adjustment in the current secondary screen browser viewport area.

[0009] In conjunction with the first aspect, in a possible implementation, the method further includes:

[0010] The main screen control module determines a first mouse position mapped in the current main screen display based on the actual height and width data of the canvas of the resized main screen image information and the actual resolution height and width data of the current cloud desktop screen, and determines a first actual mouse position mapped in the virtual machine based on the mouse position information in the image image information and the first mouse position;

[0011] The secondary screen processing module determines the second mouse position mapped in the current secondary screen display screen based on the actual height and width data of the canvas of the secondary screen image information after the size adjustment and the actual resolution height and width data, and determines the second actual mouse position mapped inside the virtual machine based on the mouse position information and the second mouse position.

[0012] In conjunction with the first aspect, in a possible implementation, the method further includes:

[0013] The main screen control module responds to the cursor shape update instruction and updates the mouse cursor shape in the current main screen display screen based on the target mouse cursor shape information carried by the cursor shape update instruction; at the same time, the secondary screen processing module updates the mouse cursor shape in the current secondary screen display screen based on the target mouse cursor shape information.

[0014] In conjunction with the first aspect, in a possible implementation, the dividing the image screen information into primary screen screen information and secondary screen screen information based on the screen configuration information includes:

[0015] In the case where the image picture information includes the starting coordinates and height and width information of the picture information boundaries of each of the multiple screens,

[0016] Determine, based on the starting coordinates, the height and width information, and the screen configuration information, a first intersection coordinate range of the current primary display screen and the image information, and a second intersection coordinate range of the current secondary display screen and the image information;

[0017] The image screen information is cut based on the first intersection coordinate range and the second intersection coordinate range, and the screens to which they belong are marked accordingly to obtain the main screen screen information and the sub-screen screen information.

[0018] In conjunction with the first aspect, in a possible implementation, the method further includes:

[0019] The main screen control module, when determining that an abnormality occurs in the current secondary screen browser tab based on the detected secondary screen browser tab connection status, regenerates a corresponding new secondary screen browser tab based on pre-stored secondary screen browser tab configuration information, and controls the opening of the new secondary screen browser tab in response to a user opening operation;

[0020] When the secondary screen processing module does not receive the main screen browser tab connection status sent by the main screen control module within a preset time period, it controls to close the corresponding current secondary screen display screen.

[0021] In conjunction with the first aspect, in a possible implementation, the method further includes:

[0022] The main screen control module and the secondary screen processing module respectively monitor input events;

[0023] The secondary screen processing module filters the monitored input events for valid information, obtains a second input event based on the filtered valid information, and sends it to the main screen control module;

[0024] The main screen control module performs corresponding format conversion processing on the monitored first input event and / or the received second input event based on the event processing request sent by the server, and then feeds back the corresponding processing results to the server.

[0025] In conjunction with the first aspect, in one possible implementation, adjusting the size of the home screen information based on a size relationship between an aspect ratio of a current home screen display screen resolution and an aspect ratio of a current home screen browser viewport area includes:

[0026] If the aspect ratio of the current main screen display screen resolution is greater than the aspect ratio of the current main screen browser viewport area, the height of the main screen image information is adjusted to a preset height value and the width is adjusted to auto;

[0027] If the aspect ratio of the current main screen display screen resolution is less than or equal to the aspect ratio of the current main screen browser viewport area, the width of the main screen image information is adjusted to a preset width value and the height is adjusted to auto.

[0028] In a second aspect, the present invention further provides a WEB terminal device, comprising a WEB terminal and a computer program stored on and executable on the WEB terminal, wherein the WEB terminal implements the multi-screen display method as described in the first aspect when executing the computer program.

[0029] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-screen display method as described in the first aspect.

[0030] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the multi-screen display method as described in the first aspect.

[0031] The embodiment of the present invention provides a multi-screen display method, a WEB terminal device, a storage medium and a program product. The main screen control module of the WEB terminal cuts the image picture information indicating that there are updates on multiple screens into main screen picture information and sub-screen picture information, and adaptively adjusts the size of the main screen picture information according to the size change of the main screen browser viewport area, and displays the main screen picture information after the size adjustment in the main screen browser viewport area with the changed size; the sub-screen processing module of the WEB terminal adaptively adjusts the size of the sub-screen picture information according to the size change of the sub-screen browser viewport area, and displays the sub-screen picture information after the size adjustment in the sub-screen browser viewport area with the changed size; in this way, the WEB terminal only needs to rely on the native support of the browser It can run on all platforms and achieve multi-screen split-screen display effects of virtual machines through browser / server (B / S architecture) without any modification and without the need to install special client software. It solves the limitations of existing technologies in multi-screen display on the WEB side, and the sizes of the updated screen information on different screens can also be adaptively adjusted as the corresponding screen browser viewport changes, ensuring that different screen information can be displayed completely and clearly under different browser viewport sizes, avoiding problems such as screen stretching and deformation, reducing server pressure and network resource consumption, and achieving accurate screen presentation on multi-screen displays. It can be widely used in multi-tasking and remote collaboration scenarios, meeting users' needs for efficient display methods, significantly improving user experience, and greatly optimizing resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0033] Figure 1 FIG1 is a flow chart of a multi-screen display method according to an embodiment;

[0034] Figure 2 A schematic diagram of the interaction between the WEB end and the server end in one embodiment;

[0035] Figure 3 This is a second flow chart of a multi-screen display method according to an embodiment;

[0036] Figure 4 is a schematic diagram of a mouse position correction process in one embodiment;

[0037] Figure 5 This is a third flow chart of a multi-screen display method according to an embodiment;

[0038] Figure 6 This is a fourth flow chart of a multi-screen display method according to an embodiment;

[0039] Figure 7FIG5 is a fifth flow chart of a multi-screen display method according to an embodiment;

[0040] Figure 8 This is a schematic diagram of the adaptive screen adjustment effect;

[0041] Figure 9 is a structural block diagram of a multi-screen display device in one embodiment;

[0042] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. In addition, the term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first" and "second" in the description and claims of the embodiments of the present invention are used to distinguish different objects, rather than to describe a specific order of objects.

[0045] With the rapid development of information technology, cloud computing and virtualization technologies have become the core infrastructure of corporate and personal work environments. Virtualization technology significantly improves resource utilization and system flexibility by creating multiple virtual machines on a single physical device. On this basis, users have an increasing demand for more efficient and flexible display methods, and multi-screen display technology has emerged. This multi-screen display technology provides users with a wider visual space and a more efficient operating experience, especially in scenarios such as multi-tasking, data comparison, and remote collaboration. For example, developers often need to cross-screen operations between code editors, real-time debuggers, and technical documentation, while financial analysts rely on multiple screens to process data in parallel for modeling and visualization. All of the above can significantly improve work efficiency.

[0046] However, in the current technical environment, the implementation of multi-screen display on cloud desktops still faces at least the following challenges:

[0047] First of all, to achieve multi-screen display of cloud desktops, users usually need to install special client software. This process of installing special client software not only increases the occupation of system resources, but also makes the use process cumbersome and requires additional installation and configuration steps. Especially when used across operating systems, it may also cause driver compatibility issues.

[0048] Secondly, for WEB users, when they access multi-screen cloud desktop virtual machines, the server defaults to using a multi-channel video stream merge transmission method, resulting in the cloud desktop page compressing and displaying all virtual screen images within a single browser viewport. This display mode seriously destroys the continuity of multi-tasking operations, fails to fully utilize the advantages of multi-screen displays, greatly limits the possibility of parallel operations, seriously affects the user experience, and thus hinders the widespread application of multi-screen technology on the WEB side.

[0049] In addition, under the existing technology, the display area is only rendered according to the preset resolution. Once the browser window size changes, the page content is difficult to adapt to the new screen size, and the cloud desktop content often overflows the visible area. At this time, the user can only rely on the scroll bar for local browsing, which is far from the user's expectation of seamless operation of the original multi-screen system.

[0050] Therefore, existing multi-screen display methods are not only cumbersome and prone to driver compatibility issues when used across operating systems, but also suffer from content overflow as the browser viewport changes, leading to significant limitations. Optimizing the display quality and user experience of multi-screen cloud desktop virtual machines on the web is crucial for improving user satisfaction and expanding the application prospects of multi-screen technology.

[0051] In order to solve the above technical problems, the present invention provides a multi-screen display method, WEB-side device, storage medium and program product, which can be run on all platforms and realize the multi-screen split-screen display effect of a virtual machine through the browser / server (B / S architecture) by only requiring modification with the native support of the browser on the WEB side and without installing special client software. The limitations of the existing technology in multi-screen display on the WEB side are solved, and the size of the to-be-updated screen information of different screens can also be adaptively adjusted as the viewport of the corresponding screen browser changes, ensuring that different screen information can be displayed completely and clearly under different browser viewport sizes, avoiding problems such as screen stretching and deformation, reducing server pressure and network resource consumption, and realizing accurate screen presentation of multi-screen display. It can be widely applied to multi-tasking and remote collaboration scenarios, meeting users' needs for efficient display methods, significantly improving user experience, and greatly optimizing resource utilization.

[0052] The following combination Figures 1 to 10 The multi-screen display method, WEB-end device, storage medium and program product of the present invention are described, wherein the multi-screen display method can be applied to the WEB end, and the WEB end includes a main screen control module and at least one sub-screen processing module, and the main screen control module interacts with the sub-screen processing module and the server for data respectively; the WEB end specifically refers to an application accessed and used through a WEB browser, such as a web page version of a WEB-end device, and the WEB-end device can be a personal computer, a laptop computer, a tablet computer, a smart phone, a smart TV and other devices. Furthermore, the multi-screen display method can also be applied to a multi-screen display device set in the WEB end, and the multi-screen display device can be implemented by software, hardware or a combination of the two. The multi-screen display method is described below by taking the execution subject of the multi-screen display method as the WEB end as an example.

[0053] To facilitate understanding of the multi-screen display method provided by the embodiments of the present invention, the multi-screen display method provided by the present invention will be described in detail below through the following exemplary embodiments. It is understood that the following exemplary embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0054] refer to Figure 1 , which is one of the flow charts of the multi-screen display method provided by an embodiment of the present invention, such as Figure 1 As shown, the multi-screen display method includes the following steps 101 to 104.

[0055] Step 101: When the main screen control module determines that the image information sent by the server indicates that multiple screens have been updated, the main screen control module divides the image information into main screen information and sub-screen information based on the screen configuration information; and when it monitors that the main screen browser viewport area changes, the main screen control module adjusts the size of the main screen information based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area.

[0056] Among them, the screen configuration information can specifically be the cloud desktop multi-screen configuration information pre-stored in the main screen control module, and the cloud desktop multi-screen configuration information is obtained after configuring the screen resolution for the main screen and all secondary screens respectively. Each screen resolution can be obtained through manual configuration by the user or return from the server.

[0057] Exemplarily, the server may be a cloud server.

[0058] It should be noted that the web client first establishes a connection with the server to facilitate subsequent data exchange with the server. In addition, in a multi-screen state, the browser tab currently communicating with the server is set as the primary screen, and the tabs on the remaining screens are opened by creating a new browser tab window. Each newly opened tab is a secondary screen.

[0059] The main screen control module is located in the main screen and is used to control the main screen. The sub-screen processing module is distributed in all sub-screens. The number of sub-screens is at least 1 and at most 3. The number of sub-screens is the same as the number of sub-screen processing modules and corresponds one to one.

[0060] In addition, in the case of multiple screens, at least one secondary screen can be generated according to the screen configuration information.

[0061] The main screen control module and each sub-screen processing module can exchange data through browser broadcast communication, and are responsible for updating and interactive processing of their respective corresponding screen images; the main screen control module can obtain the configuration information of the main screen display screen through the local storage of the main screen browser, and the sub-screen processing module can obtain the configuration information of the corresponding sub-screen display screen through the local storage of the sub-screen browser.

[0062] The browser broadcast communication method is mainly responsible for data transmission and communication management between the main screen control module and the secondary screen processing module, ensuring smooth and unimpeded information interaction between different screens.

[0063] Exemplarily, the broadcast communication method can be implemented through a broadcast channel (BroadcastChannel). BroadcastChannel is a network application programming interface (Web API) provided by the browser, which is used to implement real-time communication between browser contexts (such as different tabs, windows, iframes or frames) with the same origin (such as the same protocol, domain name, port).

[0064] Specifically, when the main screen control module establishes a connection with the server, it can receive image information sent by the server and determine whether the received screen information is image information for an update of one screen or image information for an update of at least two screens. If the screen information is image information for an update of one screen, it is further determined whether the screen update belongs to the main screen or a certain secondary screen, so as to use the screen information to fully or partially update the current main screen data or the current secondary screen data.

[0065] Conversely, if the screen image information is updated for at least two screens, then it is determined that the screen image information indicates that multiple screens have been updated; the multiple screens here can be a primary screen and at least one of all the secondary screens, or can also be at least two of all the secondary screens. The present invention is not specifically limited to this.

[0066] When the main screen control module determines that the screen image information indicates that multiple screens have been updated, it can cut the screen image information in combination with the screen configuration information to obtain the main screen image information for updating the current main screen image and at least one secondary screen image information for updating the corresponding current secondary screen image, ensuring that each screen to be updated can accurately present the corresponding screen content.

[0067] For the main screen information and at least one sub-screen information determined by the main screen control module through a splitting method, the main screen information can be used to update all screen data or part of the screen data of the main screen, and each sub-screen information can be sent to the corresponding sub-screen through a pre-set display channel to update all screen data or part of the screen data.

[0068] Exemplarily, the display channel is used to transmit data to be rendered into the canvas, ensuring that each screen can accurately obtain and display the corresponding picture content.

[0069] For the main screen control module, it is pre-set with a first monitoring application programming interface (API) for monitoring whether the main screen browser viewport area changes. In this way, when the changes in the main screen browser viewport area are monitored through the first monitoring API, the size and layout of the main screen image information can be adaptively adjusted, that is, based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area, the size of the main screen display image is adjusted; thereby ensuring that the main screen screen information can be displayed completely and clearly under different window sizes, avoiding problems such as screen stretching and deformation.

[0070] For example, when the main screen control module controls screen 1 and the secondary screen processing module processes screen 2, the following can be referred to: Figure 2 The interaction diagram between the WEB end and the server end is shown in Figure 2 In the example, tab 1 is specifically the main screen browser tab, tab 2 is specifically the secondary screen browser tab, the broadcast communication module is a module for implementing the broadcast communication method, and the server is specifically the server.

[0071] Step 102: When the secondary screen processing module detects a change in the secondary screen browser viewport area, it adjusts the size of the secondary screen image information based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area.

[0072] It should be noted that in addition to the first monitoring API pre-set in the main screen control module for monitoring whether the main screen browser viewport area has changed, each secondary screen processing module is also pre-set with a second monitoring API for monitoring whether the corresponding main screen browser viewport area has changed. In this way, when the second monitoring API monitors the change of the corresponding secondary screen browser viewport area, the size and layout of the secondary screen picture information can also be adaptively adjusted, that is, based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area, the size of the secondary screen picture information is adjusted; in this way, it is ensured that the corresponding secondary screen picture can be displayed completely and clearly under different window sizes, avoiding problems such as picture stretching and deformation.

[0073] Step 103: The main screen control module controls the main screen information after the size adjustment to be displayed in the current main screen browser viewport area.

[0074] Specifically, for the main screen information after size adjustment, that is, the main screen information after the layout and size are adaptively adjusted, the main screen control module can control it to be displayed in the current main screen browser viewport area; thereby ensuring that the main screen can accurately present the corresponding screen content.

[0075] Step 104: The secondary screen processing module displays the secondary screen image information after resizing in the current secondary screen browser viewport area.

[0076] Specifically, for the secondary screen information after size adjustment, that is, the main screen information after the layout and size are adaptively adjusted, the secondary screen processing module can display it in the corresponding current secondary screen browser viewport area; thereby ensuring that each secondary screen that stores screen updates can also accurately present the corresponding screen content.

[0077] It should be noted that the secondary screen processing module can receive the secondary screen image information sent by the main screen control module through a pre-set display channel, and for the received secondary screen image information, it first determines whether it belongs to the current secondary screen screen data. If it does, the screen is updated; if not, the screen is not updated.

[0078] In addition, it should be noted that step 103 and step 104 can be performed simultaneously or sequentially, which is not specifically limited in the present invention.

[0079] The multi-screen display method provided by the embodiment of the present invention is that the main screen control module of the WEB end cuts the image picture information indicating that there are updates on multiple screens into the main screen picture information and the sub-screen picture information, and adaptively adjusts the size of the main screen picture information according to the size change of the main screen browser viewport area, and displays the main screen picture information after the size adjustment in the main screen browser viewport area with the changed size; the sub-screen processing module of the WEB end adaptively adjusts the size of the sub-screen picture information according to the size change of the sub-screen browser viewport area, and displays the sub-screen picture information after the size adjustment in the sub-screen browser viewport area with the changed size; in this way, the WEB end only needs to make modifications with the help of the native support of the browser and does not need to install special client software, and can run on all platforms and realize the multi-screen split-screen display effect of the virtual machine through the browser / server (B / S architecture), which solves the problems of the prior art in the WEB The limitations of multi-screen display on the terminal are eliminated, and the sizes of the updated screen information on different screens can be adaptively adjusted as the browser viewport of the corresponding screen changes, ensuring that different screen information can be displayed completely and clearly in different browser viewport sizes, avoiding problems such as screen stretching and deformation, reducing server pressure and network resource consumption, and achieving accurate screen presentation on multiple screens. It can be widely used in multi-tasking and remote collaboration scenarios, meeting users' needs for efficient display methods, significantly improving user experience, and greatly optimizing resource utilization.

[0080] It is understandable that, considering that the display areas of the split main screen information and the secondary screen information are reset in a multi-screen environment, it is necessary to calibrate the mouse position to the current screen position. Based on this, in an exemplary embodiment, the multi-screen display method provided by the embodiment of the present invention may further include a mouse position calibration process, the specific process of which can be achieved in this embodiment by Figure 3 Steps 201 and 202 are implemented as shown.

[0081] Step 201: The main screen control module determines the first mouse position mapped in the current main screen display screen based on the actual height and width data of the canvas of the resized main screen image information and the actual resolution height and width data of the current cloud desktop screen, and determines the first actual mouse position mapped inside the virtual machine based on the mouse position information in the image image information and the first mouse position.

[0082] Step 202: The secondary screen processing module determines the second mouse position mapped in the current secondary screen display screen based on the actual height and width data of the canvas of the secondary screen image information after size adjustment and the actual resolution height and width data, and determines the second actual mouse position mapped inside the virtual machine based on the mouse position information and the second mouse position.

[0083] It should be noted that step 201 and step 202 may be performed simultaneously or sequentially; the present invention does not impose any specific limitation on this.

[0084] Specifically, in order to ensure that the mouse operation position on each screen is accurate in a multi-screen environment, it is necessary to accurately calibrate the mouse position while adaptively adjusting the sizes of the main screen information and the secondary screen information.

[0085] Exemplarily, the main screen control module first obtains the actual height and width data of the canvas of the main screen information after resizing and the actual resolution height and width data of the current cloud desktop screen. The specific acquisition process can obtain the actual height and width data of the canvas and the actual resolution height and width data by reading from the attribute parameters of the main screen browser; similarly, the secondary screen processing module can also obtain the actual height and width data of the canvas of the secondary screen information after resizing and the actual resolution height and width data by reading from the attribute parameters of the secondary screen browser.

[0086] Afterwards, the main screen control module can convert the mouse position from the canvas coordinate system in the main screen to the virtual coordinate system, that is, based on the actual height and width data of the canvas of the main screen information after size adjustment and the actual resolution height and width data of the current cloud desktop screen, determine the first mouse position mapped in the current main screen display screen, thereby achieving the purpose of accurate mouse positioning; similarly, the secondary screen processing module can also convert the mouse position from the canvas coordinate system in the secondary screen to the virtual coordinate system, that is, based on the actual height and width data of the canvas of the secondary screen information after size adjustment and the actual resolution height and width data of the current cloud desktop screen, determine the second mouse position mapped in the current secondary screen display screen, which can also achieve the purpose of accurate mouse positioning.

[0087] Finally, the main screen control module corrects the x-axis coordinate of the mouse in the first mouse position according to the position information of the mouse in the main screen; considering that the image information sent by the server is usually the splicing result of the main screen information to be updated and at least one sub-screen information to be updated, the x-axis coordinate of the screen before the main screen is 0. At this time, the x-axis coordinate position in the first mouse position can be used as the first actual mouse position, which is the position information of the mouse position after position correction on the current main screen.

[0088] As for the secondary screen control module, it corrects the mouse x-axis coordinate in the second mouse position according to the position information of the mouse in the secondary screen; since there is at least one main screen before the secondary screen for the image information sent by the server, the corrected mouse x-axis coordinate position is equal to the x-axis coordinate position in the second mouse position plus the sum of the x-axis coordinate positions of all screens before the current secondary screen, and the corrected mouse x-axis coordinate position is the second actual mouse position.

[0089] For example, refer to Figure 4 The schematic diagram of the mouse position correction process shown in FIG. Figure 4 The dynamic correction process of the mouse position mentioned above can ensure the global positioning accuracy of the mouse in a multi-screen environment, thereby ensuring that the multi-screen display effect is richer and higher quality.

[0090] For example, the initial position of the mouse on the main screen may be the initial position of the mouse on the main screen, and the initial position of the mouse on the main screen may be read from the attribute parameters of the main screen browser.

[0091] Specifically, the main screen control module may calculate the x-axis coordinate of the first mouse based on formula (1), and calculate the y-axis coordinate of the first mouse based on formula (2).

[0092] mousex = initial position of the mouse on the main screen / actual width of the canvas * actual width of the virtual machine (1)

[0093] In formula (1), mousex represents the x-axis coordinate of the first mouse, the actual height of the canvas is the actual width in the actual height and width data of the main screen information after size adjustment, and the actual width of the virtual machine is the actual width in the actual resolution height and width data.

[0094] mousey = initial position of the mouse on the main screen / actual height of the canvas * actual height of the virtual machine (2)

[0095] In formula (2), mousey represents the y-axis coordinate of the first mouse, the actual height of the canvas is the actual height in the actual height and width data of the main screen information after size adjustment, and the actual height of the virtual machine is the actual height in the actual resolution height and width data.

[0096] At this time, the first mouse x-axis coordinate and the first mouse y-axis coordinate can be used as the first mouse position, that is, the first mouse position is the actual mouse position after the mouse position is converted from the canvas coordinate system in the main screen to the virtual coordinate system.

[0097] It should be noted that the secondary screen processing module can also obtain the actual height and width data of the canvas of the secondary screen image information after size adjustment by reading from the attribute parameters of the secondary screen browser. Then, it can also determine the second mouse position based on the actual height and width data of the canvas of the secondary screen image information after size adjustment and the actual resolution height and width data of the current cloud desktop screen. The determination process can refer to the determination process of the first mouse position.

[0098] For example, when the initial position of the main screen mouse in formula (1) is replaced by the initial position of the secondary screen mouse, and the actual width of the canvas is replaced by the actual width in the actual height and width data of the secondary screen image information after the size is adjusted, the second mouse x-axis coordinate can be calculated; and when the initial position of the main screen mouse in formula (2) is replaced by the initial position of the secondary screen mouse, and the actual height of the canvas is replaced by the actual height in the actual height and width data of the secondary screen image information after the size is adjusted, the second mouse y-axis coordinate can be calculated; then, the second mouse x-axis coordinate and the second mouse y-axis coordinate are used as the second mouse position, which is the actual position of the mouse after the mouse position is converted from the canvas coordinate system in the secondary screen to the virtual coordinate system.

[0099] It is understandable that, considering that in a multi-screen environment, the shape, position, and visibility of the mouse cursor can be updated in a timely manner based on the display content and user operations of different screens to maintain the consistency and accuracy of the mouse cursor in the multi-screen environment. Based on this, in an exemplary embodiment, the multi-screen display method provided by the embodiment of the present invention can also include a mouse cursor update process, the specific process of which can be implemented in this embodiment through the following steps.

[0100] The main screen control module responds to the cursor shape update instruction and updates the mouse cursor shape in the current main screen display screen based on the target mouse cursor shape information carried by the cursor shape update instruction; at the same time, the sub-screen processing module updates the mouse cursor shape in the current sub-screen display screen based on the target mouse cursor shape information.

[0101] The cursor shape update instruction may be an instruction generated when an operation of updating the mouse cursor shape is detected.

[0102] Specifically, when the main screen control module detects an operation to update the mouse cursor shape, it can update the cursor shape information in the current main screen display screen, and send the target mouse cursor shape information that needs to be updated to the sub-screen processing module through the cursor channel. After the sub-screen processing module receives the target mouse cursor shape information, it can synchronously update the mouse cursor shape in the current sub-screen display screen.

[0103] Exemplarily, the cursor channel is used to transmit data such as the shape, position, and visibility status of the mouse pointer to ensure the consistency and accuracy of the mouse cursor between different screens.

[0104] In this way, when the main screen control module updates the mouse cursor shape in the current main screen display, the target mouse cursor shape information is transmitted through the cursor channel and synchronously updated to the secondary screen display to ensure that the mouse cursor shapes of the main screen and the secondary screen are consistent; thereby meeting the user's diverse needs for efficient display methods.

[0105] Based on the above Figure 1 In the method shown in FIG. 1 , in an exemplary embodiment, in step 101, the image screen information is divided into the main screen screen information and the sub-screen screen information based on the screen configuration information. The specific process in this embodiment can be achieved by Figure 5 Steps 301 and 302 are implemented as shown.

[0106] Step 301: When the image screen information includes the starting coordinates and height and width information of the boundaries of multiple screens, determine the first intersection coordinate range of the current main screen display screen and the image screen information, and the second intersection coordinate range of the current secondary screen display screen and the image screen information based on the starting coordinates, the height and width information, and the screen configuration information.

[0107] Step 302: Cut the image screen information based on the first intersection coordinate range and the second intersection coordinate range and mark the corresponding screens to obtain the main screen screen information and the sub-screen screen information.

[0108] Specifically, considering that the SPICE protocol usually uses an incremental update method to update the screen, and each undivided data returned contains multiple screens, the main screen control module needs to divide the image screen information by screen after obtaining the screen information that needs to be updated (that is, receiving the image screen information sent by the server), and then update it to the corresponding display screen canvas respectively.

[0109] As for the image screen information segmentation process, the starting coordinates and height and width information of multiple screens can be compared with the screen sizes of the current main screen display screen and the corresponding current sub-screen display screen, respectively, and the overlapping area can be quickly determined through the coordinate extreme value comparison process, that is, the first intersection coordinate range of the current main screen display screen and the image screen information, and the second intersection coordinate range of the current sub-screen display screen and the image screen information can be determined.

[0110] Exemplarily, for the boundaries of the main screen information in the image information, the boundaries of the main screen information can be compared with the screen boundaries of the main display screen in the screen configuration information, and the larger value in the left boundary is used as the left boundary of the intersection, the smaller value in the right boundary is used as the right boundary of the intersection, the larger value in the upper boundary is used as the upper boundary of the intersection, and the larger value in the lower boundary is used as the lower boundary of the intersection. Then, it is determined whether the right boundary of the intersection is strictly greater than the left boundary of the intersection and whether the lower boundary of the intersection is strictly greater than the upper boundary of the intersection. If the right boundary of the intersection is strictly greater than the left boundary of the intersection and the lower boundary of the intersection is strictly greater than the upper boundary of the intersection, it is determined that there is a valid intersection between the current main display screen and the image information, and the coordinate range of the valid intersection is used as the first intersection coordinate range; similarly, the second intersection coordinate range can be obtained.

[0111] At this time, the main screen control module can cut the image screen information based on the first intersection coordinate range and the second intersection coordinate range, and mark the screen to which it belongs, to obtain the main screen screen information and the secondary screen screen information. In this way, when the secondary screen processing module receives the secondary screen screen information transmitted by the display channel, it can first determine whether it belongs to the current screen data based on its screen mark. If it does, the screen will be updated; otherwise, the screen will not be updated.

[0112] It should be noted that image information usually has the following three formats: 1. imageData: directly operate pixel data; 2. Canvas data (canvas data): use the getImageData method; 3. img: need to be drawn on the Canvas first, and then operate the data; the final converted data are all in imageData format.

[0113] In addition, if the image information is only on one display screen, there are two cases:

[0114] Case 1): If the image information is all main screen information, update it directly;

[0115] Case 2) The image information is all secondary screen information. Some format data cannot be sent through the display channel, so format conversion is required. The specific transmission process includes the following three methods:

[0116] Method 1) imageData: can be transmitted directly through the display channel;

[0117] Method 2) Canvas: cannot be transmitted directly and needs to be converted into data formats such as imageData before transmission;

[0118] Method 3) cannot be transmitted directly. The image data of the img element needs to be converted into a Base64 string or Uint8Array format and then transmitted through the display channel.

[0119] It is understandable that, considering that screen tabs are often accidentally closed or refreshed, the main screen control module is required to maintain its own screen data and regenerate a new secondary screen tab when an abnormality occurs in the secondary screen tab; at the same time, the secondary screen processing module is also required to automatically close the corresponding secondary screen if it does not receive the information sent by the main screen control module for a long time. Based on this, in an exemplary embodiment, the multi-screen display method provided by the embodiment of the present invention may also include a main and secondary screen management and heartbeat monitoring process, the specific process of which can be achieved through Figure 6 Steps 401 and 402 are implemented as shown.

[0120] Step 401: When the main screen control module determines that an abnormality has occurred in the current secondary screen browser tab based on the detected secondary screen browser tab connection status, it regenerates a corresponding new secondary screen browser tab based on the pre-stored secondary screen browser tab configuration information, and controls the opening of the new secondary screen browser tab in response to the user's opening operation.

[0121] Step 402: If the secondary screen processing module does not receive the primary screen browser tab connection status sent by the primary screen control module within a preset time period, the secondary screen processing module controls closing the corresponding current secondary screen display screen.

[0122] It should be noted that step 401 and step 402 may be performed simultaneously or sequentially; the present invention does not impose any specific limitation on this.

[0123] Specifically, for the main screen control module, it can check the connection status of the secondary screen browser tab, update the main and secondary screen configuration information, and manage the secondary screen browser tab. If the current secondary screen browser tab is accidentally closed or refreshed, the main screen control module will not receive the secondary screen heartbeat message for a long time. At this time, a prompt message will be displayed on the current main screen display screen asking whether to generate a new secondary screen browser tab. The user enters the new secondary screen browser tab to determine the generation instruction, regenerate the corresponding new secondary screen browser tab, and then open the new secondary screen browser tab through the user input of the open instruction. If the user enters the new secondary screen browser tab to cancel the generation instruction in response to the above prompt information, the initially received image screen information will be merged into one screen screen information, and the main and secondary screens will no longer be distinguished.

[0124] For the secondary screen control module, it can check the connection status of the main screen browser tab and set a timeout period. If it fails to receive the main screen browser tab connection status sent through the heartbeat channel for a long time, it will automatically close the corresponding current secondary screen display screen. For example, the heartbeat channel is used to check whether the status of the main and secondary screen tabs is normal.

[0125] In this way, through the above-mentioned main and secondary screen management and heartbeat monitoring process, the communication purpose between different browser tabs can be achieved, ensuring the consistency and coherence of multi-screen display.

[0126] It is understandable that, considering that in a multi-screen environment, all input events of the main and secondary screens can be monitored, and the monitored input events are uniformly processed and sent to the server, so as to achieve the purpose of collaborative processing and normalized management of multi-screen input events. Based on this, in an exemplary embodiment, the multi-screen display method provided by the embodiment of the present invention can also include an input event processing process, the specific process of which can be achieved in this embodiment through Figure 7 Steps 501 to 503 are implemented as shown.

[0127] Step 501: The main screen control module and the secondary screen processing module monitor input events respectively.

[0128] Step 502: The secondary screen processing module filters the monitored input events for valid information, obtains a second input event based on the filtered valid information, and sends it to the main screen control module.

[0129] Step 503: The main screen control module performs corresponding format conversion processing on the monitored first input event and / or the received second input event based on the event processing request sent by the server, and then feeds back the corresponding processing result to the server.

[0130] Specifically, the main screen control module and the sub-screen processing module can respectively set up event listeners with input event monitoring functions, and each event listener is used to continuously monitor input events on the screen; that is, the main screen control can monitor main screen input events, and the sub-screen processing module can also monitor sub-screen input events. Both the main screen input events and the sub-screen input events can include one of the mouse events and keyboard events.

[0131] When the secondary screen processing module monitors an input event, it first filters the valid information; the valid information here may include the specific position information of the mouse and the mouse press position information, or the specific key information of the keyboard and the number of presses; then it further generates an event based on the filtered valid information to obtain a second input event, and then sends the second input event to the main screen control module through the input channel.

[0132] For example, the input channel is used to transmit events of input devices such as keyboard and mouse, including coordinate position (XY axis), key status and other data, to achieve multi-screen collaborative operation.

[0133] At this time, the main screen control module can perform corresponding format conversion processing based on the monitored first input event and / or the received second input event, and then feed back the processing results to the server; thereby achieving the purpose of collaborative processing and normalized management of multi-screen input events.

[0134] Exemplarily, in the secondary screen processing module, the user's operations on the current secondary screen display screen (mouse and keyboard events) are monitored; then, after monitoring the execution of the operation, valid information is obtained (such as: mouse movement only requires the mouse position offsetX and offsetY, and mouse press only requires the specific button pressed by the mouse), and the second input event is sent to the main screen control module through the input channel; finally, the main screen control module calls different methods according to different input events, and then sends them to the server.

[0135] In this way, through the above-mentioned input event cross-display screen processing process, the purpose of seamless interaction of input device events such as keyboard and mouse between different display screens can be achieved, ensuring that user operations on the display screen can be accurately identified and responded to in a timely manner.

[0136] Based on the above Figure 1 In the method shown, in an exemplary embodiment, in step 101, the size of the main screen information is adjusted based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area. The specific process can be implemented in this embodiment through the following steps.

[0137] If the aspect ratio of the current main screen display screen resolution is greater than the aspect ratio of the current main screen browser viewport area, the height of the main screen screen information is adjusted to the preset height value and the width is adjusted to auto; if the aspect ratio of the current main screen display screen resolution is less than or equal to the aspect ratio of the current main screen browser viewport area, the width of the main screen screen information is adjusted to the preset width value and the height is adjusted to auto.

[0138] Exemplarily, the preset height value and the preset width value may both be 100%.

[0139] It should be noted that, considering that the aspect ratio of the display screen resolution and the aspect ratio of the display screen canvas are generally the same, the main screen control module can adjust the width and height of the main screen image information based on the relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area. Alternatively, the width and height of the main screen image information can be adjusted based on the relationship between the aspect ratio of the current main screen display screen canvas and the aspect ratio of the current main screen browser viewport area.

[0140] Since, when the canvas is initialized, the canvas's attribute parameters such as width and height are set, and the size of the display area is updated according to these attribute parameters during rendering; therefore, after the canvas is initialized, the height and width of the main screen information are modified according to the above attribute parameters and the changes in the browser viewport area monitored through the browser's native API.

[0141] For example, if the aspect ratio of the current home screen display canvas is greater than the height-to-width ratio of the current home screen browser viewport area, the width of the home screen information is set to 100% and the height is set to auto. If the aspect ratio of the current home screen display canvas is less than or equal to the height-to-width ratio of the current home screen browser viewport area, the height of the home screen information is set to 100% and the width is set to auto. Furthermore, throttling can be used to reduce the number of events processed by the browser within a unit event, thereby reducing performance consumption.

[0142] In addition, it should be noted that the secondary screen processing module can also adjust the width and height of the secondary screen image information based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area. Alternatively, the width and height of the secondary screen image information can also be adjusted based on the size relationship between the aspect ratio of the current secondary screen display screen canvas and the aspect ratio of the current secondary screen browser viewport area.

[0143] For example, if the aspect ratio of the current secondary screen display screen canvas (or the aspect ratio of the current secondary screen display screen resolution) is greater than the aspect ratio of the current secondary screen browser viewport area, the width of the secondary screen image information is set to 100% and the height is auto; if the aspect ratio of the current secondary screen display screen canvas (or the aspect ratio of the current secondary screen display screen resolution) is less than or equal to the aspect ratio of the current secondary screen browser viewport area, the height of the secondary screen image information is set to 100% and the width is auto. For details, please refer to Figure 8 Schematic diagram of the adaptive picture adjustment effect shown.

[0144] In the multi-screen display method provided by the embodiment of the present invention, the main screen control module is mainly responsible for establishing a connection with the server and receiving and sending data, cutting and processing the received image, updating the main screen picture, sending the processed secondary screen image data, managing the tabs of the main screen and the secondary screen, and synchronizing the main and secondary screen operations and cursors to ensure the synchronization and consistency of multi-screen display; the secondary screen processing module is mainly responsible for the screen update, input event monitoring and processing, and cursor synchronization of the secondary screen to ensure the normal operation of the secondary screen and the user interaction experience, realize the multi-screen display function based on the WEB cloud desktop, significantly improve the user experience, and with the native support of the browser, it can run on all platforms without installing dedicated client software, further optimizes resource utilization, is suitable for multi-tasking and remote collaboration scenarios, and can meet the user's demand for efficient display methods.

[0145] It should be noted that although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0146] In one embodiment, the present invention further provides a multi-screen display device, such as Figure 9 As shown, the multi-screen display device includes: a display screen splitting unit 901, a screen information adjusting unit 902 and a multi-screen display unit 903.

[0147] The display screen splitting unit 901 is configured to split the image screen information into main screen information and sub-screen information based on the screen configuration information when the main screen control module determines that the image screen information sent by the server indicates that multiple screens have been updated.

[0148] The screen information adjustment unit 902 is configured to adjust the size of the main screen screen information based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area when the main screen control module monitors changes in the main screen browser viewport area; and to adjust the size of the secondary screen screen information based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area when the secondary screen browser viewport area is monitored changes.

[0149] The multi-screen display unit 903 is configured to control the main screen information after size adjustment by the main screen control module and display it in the current main screen browser viewport area, and the secondary screen processing module controls the secondary screen information after size adjustment to display it in the current secondary screen browser viewport area.

[0150] In one embodiment, the multi-screen display device provided by the present invention may also include a mouse position correction unit, which is configured to determine the first mouse position mapped in the current main screen display screen based on the actual height and width data of the canvas of the main screen image information after size adjustment and the actual resolution height and width data of the current cloud desktop screen by the main screen control module, and determine the first actual mouse position mapped inside the virtual machine based on the mouse position information in the image image information and the first mouse position; the secondary screen processing module determines the second mouse position mapped in the current secondary screen display screen based on the actual height and width data of the canvas of the secondary screen image information after size adjustment and the actual resolution height and width data, and determines the second actual mouse position mapped inside the virtual machine based on the mouse position information and the second mouse position.

[0151] In one embodiment, the multi-screen display device provided by the present invention may also include a mouse cursor update unit, which is configured for the main screen control module to respond to the cursor shape update instruction and update the mouse cursor shape in the current main screen display screen based on the target mouse cursor shape information carried by the cursor shape update instruction; at the same time, the secondary screen processing module updates the mouse cursor shape in the current secondary screen display screen based on the target mouse cursor shape information.

[0152] In one embodiment, the display screen segmentation unit 901 is configured to determine, when the image screen information includes the starting coordinates and height and width information of the screen information boundaries of multiple screens, a first intersection coordinate range of the current main screen display screen and the image screen information, and a second intersection coordinate range of the current sub-screen display screen and the image screen information based on each starting coordinate, each height and width information and screen configuration information; based on the first intersection coordinate range and the second intersection coordinate range, the image screen information is cut and the corresponding screens are marked to obtain the main screen screen information and the sub-screen screen information.

[0153] In one embodiment, the multi-screen display device provided by the present invention may also include a main-sub-screen management and heartbeat monitoring unit, which is configured to regenerate a corresponding new sub-screen browser tab based on pre-stored sub-screen browser tab configuration information when the main screen control module determines that the current sub-screen browser tab is abnormal based on the detected sub-screen browser tab connection status, and control the opening of the new sub-screen browser tab in response to the user's opening operation; when the sub-screen processing module does not receive the main screen browser tab connection status sent by the main screen control module within a preset time length, control the closing of the corresponding current sub-screen display screen.

[0154] In one embodiment, the multi-screen display device provided by the present invention may further include an input event processing unit, which is configured for the main screen control module and the sub-screen processing module to respectively monitor input events; the sub-screen processing module filters the effective information of the monitored input events, and obtains the second input event based on the filtered effective information and sends it to the main screen control module; the main screen control module performs corresponding format conversion processing on the monitored first input event and / or the received second input event based on the event processing request issued by the server, and then feeds back the corresponding processing results to the server.

[0155] In one embodiment, the screen information adjustment unit 902 is configured to adjust the height of the main screen screen information to a preset height value and the width to auto if the aspect ratio of the current main screen display screen resolution is greater than the aspect ratio of the current main screen browser viewport area; if the aspect ratio of the current main screen display screen resolution is less than or equal to the aspect ratio of the current main screen browser viewport area, the width of the main screen screen information is adjusted to a preset width value and the height is adjusted to auto.

[0156] An embodiment of the present invention further provides a WEB terminal device, including a WEB terminal and a computer program stored on the WEB terminal and executable on the WEB terminal. When the WEB terminal executes the computer program, the aforementioned multi-screen display method is implemented.

[0157] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the aforementioned multi-screen display method when executed by a processor.

[0158] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the aforementioned multi-screen display method when executed by a processor.

[0159] Reference below Figure 10 , which shows a schematic structural diagram of a computer system 1000 of a terminal device or server suitable for implementing an embodiment of the present invention.

[0160] like Figure 10 As shown, computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of system 1000 are also stored in RAM 1003. CPU 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0161] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read from the removable media can be installed in the storage section 1008 as needed.

[0162] In particular, according to the embodiments of the present disclosure, the above reference Figure 1The described processes may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising instructions for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009 and / or installed from the removable medium 1011.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] The units or modules described in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. The names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0165] As another aspect, the present invention further provides a computer-readable storage medium. This computer-readable storage medium may be included in the apparatus described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the multi-screen display method described in the present invention.

[0166] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

Claims

1. A multi-screen display method, characterized in that: Applied to a WEB end, the WEB end includes a main screen control module and at least one secondary screen processing module, the main screen control module exchanges data with the secondary screen processing module and the server respectively, the main screen control module controls the main screen browser tab, the secondary screen processing module processes the secondary screen browser tab, and the secondary screen browser tab and the secondary screen browser tab both belong to the WEB end; the method includes: The main screen control module, when determining that the image information sent by the server indicates that multiple screens have been updated, divides the image information into main screen information and sub-screen information based on the screen configuration information; and, when monitoring a change in the main screen browser viewport area, adjusts the size of the main screen information based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area; When the secondary screen processing module detects a change in the secondary screen browser viewport area, the secondary screen processing module adjusts the size of the secondary screen image information based on the size relationship between the aspect ratio of the current secondary screen display screen resolution and the aspect ratio of the current secondary screen browser viewport area; The main screen control module controls the main screen image information after size adjustment to be displayed in the current main screen browser viewport area; The secondary screen processing module displays the secondary screen image information after size adjustment in the current secondary screen browser viewport area; The main screen control module determines a first mouse position mapped in the current main screen display based on the actual height and width data of the canvas of the resized main screen image information and the actual resolution height and width data of the current cloud desktop screen, and determines a first actual mouse position mapped in the virtual machine based on the mouse position information in the image image information and the first mouse position; The secondary screen processing module determines the second mouse position mapped in the current secondary screen display screen based on the actual height and width data of the canvas of the secondary screen image information after the size adjustment and the actual resolution height and width data, and determines the second actual mouse position mapped inside the virtual machine based on the mouse position information and the second mouse position.

2. The method according to claim 1, characterized in that The method further comprises: The main screen control module responds to the cursor shape update instruction and updates the mouse cursor shape in the current main screen display screen based on the target mouse cursor shape information carried by the cursor shape update instruction; at the same time, the secondary screen processing module updates the mouse cursor shape in the current secondary screen display screen based on the target mouse cursor shape information.

3. The method according to any one of claims 1 to 2, characterized in that The step of dividing the image screen information into main screen screen information and sub-screen screen information based on the screen configuration information includes: In the case where the image picture information includes the starting coordinates and height and width information of the picture information boundaries of each of the multiple screens, Determine, based on the starting coordinates, the height and width information, and the screen configuration information, a first intersection coordinate range of the current primary display screen and the image information, and a second intersection coordinate range of the current secondary display screen and the image information; The image screen information is cut based on the first intersection coordinate range and the second intersection coordinate range, and the screens to which they belong are marked accordingly to obtain the main screen screen information and the sub-screen screen information.

4. The method according to any one of claims 1 to 2, characterized in that The method further comprises: The main screen control module, when determining that an abnormality occurs in the current secondary screen browser tab based on the detected secondary screen browser tab connection status, regenerates a corresponding new secondary screen browser tab based on pre-stored secondary screen browser tab configuration information, and controls the opening of the new secondary screen browser tab in response to a user opening operation; When the secondary screen processing module does not receive the main screen browser tab connection status sent by the main screen control module within a preset time period, it controls to close the corresponding current secondary screen display screen.

5. The method according to any one of claims 1 to 2, characterized in that The method further comprises: The main screen control module and the secondary screen processing module respectively monitor input events; The secondary screen processing module filters the monitored input events for valid information, obtains a second input event based on the filtered valid information, and sends it to the main screen control module; The main screen control module performs corresponding format conversion processing on the monitored first input event and / or the received second input event based on the event processing request sent by the server, and then feeds back the corresponding processing results to the server.

6. The method according to any one of claims 1 to 2, characterized in that The adjusting the size of the main screen image information based on the size relationship between the aspect ratio of the current main screen display screen resolution and the aspect ratio of the current main screen browser viewport area includes: If the aspect ratio of the current main screen display screen resolution is greater than the aspect ratio of the current main screen browser viewport area, the height of the main screen image information is adjusted to a preset height value and the width is adjusted to auto; If the aspect ratio of the current main screen display screen resolution is less than or equal to the aspect ratio of the current main screen browser viewport area, the width of the main screen image information is adjusted to a preset width value and the height is adjusted to auto.

7. A web terminal device, comprising a web terminal and a computer program stored on the web terminal and operable on the web terminal, characterized in that: When the WEB end executes the computer program, the multi-screen display method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-screen display method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-screen display method according to any one of claims 1 to 6 is implemented.

Images