Multi-application window projection screen management system, medium and program product

The multi-application window projection management system addresses complex operations and slow responses by distinguishing primary and secondary devices, generating secure projection instructions, and optimizing content transmission, resulting in improved user experience and security.

CN119071552APending Publication Date: 2024-12-03GUANGZHOU LANGO ELECTRONICS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411175722.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing screen projection technologies face complex operations and slow responses when switching between multiple applications and projection windows, affecting user experience.

Method used

A multi-application window projection management system that distinguishes between primary and secondary devices, generates and verifies screen projection instructions based on window ID and activity information, and transmits content efficiently to reduce delays and enhance security.

Benefits of technology

Facilitates convenient and fluent switching between multiple devices and application windows, reduces delays, and enhances security by ensuring privacy and preventing unauthorized access to projection content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071552A_ABST
    Figure CN119071552A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-application-window projection screen management system, a medium and a program product, relates to the technical field of projection screen management, and solves the technical problems of complex operation and slow response in a use process in an application scene in which a projection screen is frequently required to be called. Comprising a window management module, a screen projection management module and a touch control processing module, according to the invention, the plurality of devices to be subjected to screen projection are subjected to primary and secondary distinguishing, so that when the screen projection contents of the plurality of devices are switched, the primary projection device can be primary, and when the plurality of subsequent devices to be subjected to screen projection are switched, the devices can be quickly switched back to the primary projection device; a large amount of operation time spent when other devices to be projected are switched to display the content needing screen projection is avoided, the touch control processing module is matched to switch the multiple devices to be projected and the multiple application windows for screen projection, operation is convenient, delay can be reduced, and smoothness is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of screen projection and relates to touch control technology, and specifically is a multi-application window screen projection management system, medium and program product. Background Art

[0002] With the popularity of mobile devices and smart display devices, people's demand for screen projection technology continues to grow. From office meetings to home entertainment, from professional presentations to distance education, screen projection technology is widely used in all aspects of life.

[0003] Although the existing screen projection technology can project screen contents, it is often necessary to operate and set up the projection system again when switching between multiple applications and projection windows. In application scenarios where screen projection is often required, the operation is complicated during use, and there is a certain delay in data transmission to the projection device, resulting in slow response. This directly affects people's user experience. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a multi-application window projection management system, medium and program product, which are used to solve the technical problems of complex operation and slow response during use in application scenarios where screen projection is often required. The present invention distinguishes between primary and secondary devices to be projected, so that when the projection content of multiple devices is switched, the main projection device can be the primary, and the generated projection content is transmitted to the projection device in conjunction with the matching of the window ID and the activity information in the window list and the primary and secondary information to solve the above-mentioned problem.

[0005] To achieve the above-mentioned object, a first aspect of the present invention provides a multi-application window projection management system, comprising: a window management module, a projection management module and a touch processing module;

[0006] The window management module: identifies multiple application windows on multiple devices to be projected; manages the multiple application windows on the devices to be projected according to the operation instructions; and

[0007] Get the activity information of the current window on the device to be projected and store it in the window list; the activity information includes the window ID, resolution, location and application name of the window;

[0008] The screen projection management module: distinguishes the devices to be projected into primary and secondary and generates and sends screen projection instructions based on activity information; is used to input the screen projection instructions to the screen projection device and verify them; according to the verification results, the primary and secondary information and window ID of the devices to be projected are sent to the screen projection device; wherein the primary and secondary information includes the primary projection device and the secondary projection device; and,

[0009] Match the window ID obtained by the projection device with the activity information in the window list, and transmit the generated projection content to the projection device according to the matching result and primary and secondary information;

[0010] The touch control processing module: processes the operation instructions of the user on the screen projection device after obtaining the operation instructions; wherein the operation instructions include interaction information and sub-switching commands; and,

[0011] Control the device to be projected or the projected device that has been verified according to the operation instructions.

[0012] Preferably, obtaining the activity information of the current window on the device to be projected includes:

[0013] Get the number of windows running on the device to be projected and mark it as n; mark the windows according to the running time of each of the n windows to obtain the window ID;

[0014] Count the resolutions occupied by n windows in the device to be projected, and mark them as xk×yk; where n is a positive integer, x represents the number of pixels occupied by the window horizontally, y represents the number of pixels occupied vertically, and k∈[1,n];

[0015] Establish an xy plane with the center point of the device to be projected as the origin and pixels as the unit, obtain the coordinates of the upper left corners of several windows, and mark them as the occupied positions;

[0016] Extract the application names of several running windows through Python.

[0017] Preferably, the step of distinguishing the devices to be projected into primary and secondary devices and generating and sending a projection instruction based on the activity information includes:

[0018] Mark the windows to be projected on several devices as projection windows, select one device to be projected from the devices to be projected as the primary projection device, and mark the other devices to be projected as secondary projection devices;

[0019] The activity information corresponding to the primary projection device and the secondary projection device are obtained respectively, and several projection instructions are generated based on the corresponding activity information and the type of the device to be projected, and the corresponding projection instructions are sent to the verification end through the corresponding device to be projected; wherein the type of the device to be projected includes the primary projection device and the secondary projection device.

[0020] Preferably, the generating of several screen projection instructions based on the corresponding activity information and the type of the device to be projected includes:

[0021] Extract the type of the device to be projected, mark the projection window corresponding to the primary projection device in the device to be projected with features, and mark the projection window corresponding to the secondary projection device in the device to be projected with non-features;

[0022] The window ID, resolution, location and application name corresponding to each projection window are extracted and hashed to obtain a hash value; a combination of n consecutive letters and numbers is randomly selected from the hash value and marked as a projection instruction.

[0023] It should be noted that the characteristic mark can be the number 0, and the non-characteristic mark can be a non-zero number.

[0024] The screen projection instructions generated in this way are random, and the same screen projection instructions cannot be easily obtained. In addition, the window position and other factors are necessarily different each time the screen is projected. Therefore, even if the screen projection content is the same, the screen projection instructions will change. In addition, the instructions for each device to be projected are different, which ensures that the screen projection instructions cannot be easily cracked, and can prevent the screen projection instructions from being leaked. There are other interfering devices projecting on the screen projection device, and it is impossible to reversely obtain the original activity information.

[0025] Preferably, the step of inputting a screen projection instruction to the screen projection device and verifying the instruction includes:

[0026] Enter the screen projection command in the screen projection device, and the screen projection device sends the screen projection command to the verification end;

[0027] The verification end determines whether the screen projection instruction sent by the device to be projected is the same as the screen projection instruction sent by the projection device; if yes, the verification result of the device to be projected is marked as passed verification; if not, the verification result of the device to be projected is marked as verification failure.

[0028] Preferably, the step of sending the primary and secondary information and the window ID of the device to be projected to the projection device according to the verification result includes:

[0029] Determine whether the verification result of the device to be projected is passed; if yes, extract the primary and secondary information of the device to be projected and the window ID that needs to be projected, and send the primary and secondary information and window ID of the device to be projected to the projection device; if no, input the projection command to the projection device and verify it.

[0030] Preferably, the generating screen projection content and transmitting it to the screen projection device according to the matching result and the primary and secondary information includes:

[0031] Obtain the content displayed by the matched window ID on the main projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a main projection data packet; transmit the main projection data packet to the projection device;

[0032] Obtain the content displayed by the matched window ID on the secondary projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a secondary projection data packet;

[0033] Extract the file information of the secondary projection data packet and the transmission speed between the secondary projection device and the screen projection device; cut the secondary projection data packet according to the file information and the transmission speed to obtain several secondary projection sub-data packets; arrange the several secondary projection sub-data packets and transmit them to the screen projection device; wherein the file information is the file size of the secondary projection data packet.

[0034] Preferably, the cutting of the secondary data packet according to the file information and the transmission speed includes:

[0035] The file size and transmission speed in the file information are extracted and marked as S and V respectively, and the cutting quantile d is calculated by the formula d=V×t / S; where t is the preset transmission time;

[0036] The secondary data packet is cut based on the cutting quantile point to obtain two primary cut data packets;

[0037] The integrity of the two initial data packets is verified; when the two initial data packets are complete, the initial data packets are marked as secondary sub-data packets; when the two initial data packets are incomplete, the two initial sub-data packets are corrected by supplementing and eliminating them respectively to obtain secondary sub-data packets.

[0038] Furthermore, when the second sub-data packet is still large, it can be cut again, and when the two initial cut sub-data packets are corrected, a check field can be added before the data packet is cut, and the data to be supplemented or removed can be determined according to the position of the check field, or an existing network communication protocol framework, such as TCP / IP, UDP, etc., can be used;

[0039] It should be noted that the unit of S is MB, and the unit of V is MB / s. When the sub-data packet is divided into bits, it is ensured that each sub-data contains necessary meta-information so that it can be correctly reassembled at the receiving end. When the meta-information cut according to the cutting point is incomplete, the incomplete information can be selected to be removed or supplemented, so as to ensure the integrity of the content in the subsequent screen projection.

[0040] Preferably, the step of arranging the plurality of projection sub-data packets into messages and transmitting them to the projection device includes:

[0041] Based on the sequential marking i of the secondary devices that have passed the verification, the secondary sub-data packets corresponding to the secondary devices are marked as Bij according to the composition order of the data packets; wherein Bij is the jth data packet cut from the secondary data packet in the i-th secondary device; wherein i and j are both positive integers;

[0042] Arrange several sub-projection data packets in a transmission order specified by the transmission rule to obtain a data queue; and transmit several sub-projection data packets to the screen projection device according to the data queue.

[0043] Preferably, the transmission order specified by the transmission rule includes:

[0044] Generate a matrix [Bij] with the labels Bij corresponding to the sub-data packets of the multiple times, and arrange the labels Bij in the order of the column vectors from left to right to obtain the label order;

[0045] The plurality of sub-data packets corresponding to the marks are sorted according to the mark order to obtain the transmission order.

[0046] Preferably, the processing after obtaining the operation instruction of the user on the screen projection device includes:

[0047] Step 1: extracting the interaction information of the interactive device; wherein the interaction information is the indicated coordinates, timestamp, control instructions and moving direction of the indicated point of the interactive device;

[0048] Step 2: Determine whether the interactive information is a basic instruction of the interactive device; if yes, send the interactive information to the corresponding primary projection device or secondary projection device; if no, jump to step 3; wherein the basic instruction is the basic operation of the interactive device;

[0049] Step three: Determine whether there is a match between the interactive information and the screen projection switching instruction library; if so, switch the main projection device and the secondary projection device according to the sub-switching command of the matching result; if not, do not perform any operation.

[0050] It should be noted that the interactive device may be a mouse, a laser pen, a camera or other devices.

[0051] Preferably, the sub-switching commands in the screen projection switching instruction library are set based on the interactive information;

[0052] The sub-switching command includes a primary-secondary device switching command, a secondary-secondary device switching command, a previous window switching command and a next window switching command.

[0053] Preferably, the sub-switching command is set based on the interactive information, including:

[0054] Extract the indicated coordinates of the indicated point of the interactive device, and set a preset central rectangle of Chi Xun with the indicated coordinates as the center, and extend the line segments along the four right angles intersecting the line segments from the four right angles of the central rectangle as the starting points to obtain the demarcation line;

[0055] Mark the four areas enclosed by two adjacent demarcation lines and the central rectangular boundary as the operation termination area;

[0056] The operations of moving the pointer coordinates with control instructions from the central rectangle to the four operation termination areas within the preset time are marked as the primary-secondary device switching command, the secondary-secondary device switching command, the previous window switching command and the next window switching command respectively.

[0057] Switching between multiple devices to be projected and multiple application windows in the above manner is not only more convenient to operate, but also can reduce delays and increase fluency.

[0058] Preferably, the controlling the screen projection device or the screen projection device that has been verified according to the operation instruction includes:

[0059] S1: Determine whether the operation instruction is interactive information; if yes, control the verified device to be projected to which the current display window on the projection device belongs according to the interactive information; if no, jump to S2;

[0060] S2: Determine whether the interactive information is a primary-to-secondary device switching command or a secondary-to-secondary device switching command; if yes, call the primary data packet or secondary data packet corresponding to the verified device to be projected through the projection device, and display it on the projection device; if no, control the verified device to be projected to which the current display window on the projection device belongs to perform the corresponding window switching.

[0061] It should be noted that the verified devices to be projected include the primary projection device and the secondary projection device.

[0062] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-application window projection management system as described in the first aspect above.

[0063] In a third aspect, the present invention further provides a computer program product, which is used to run on a computer and / or a terminal device to implement the multi-application window projection management system as described in the first aspect above.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. In the present invention, by distinguishing between primary and secondary devices to be projected, the primary projection device can be used as the primary when switching the projection content of multiple devices. When switching between the subsequent multiple devices to be projected, the device can be quickly switched back to the primary projection device, avoiding a large amount of operation time when switching from other devices to be projected to display the content to be projected. In addition, cooperating with the touch processing module to switch between the multiple devices to be projected and between the multiple application windows is not only more convenient to operate, but also can reduce delays and increase fluency.

[0066] 2. In the present invention, by obtaining the window activity information of the device to be projected, the screen is projected with the content of the window during subsequent projection, thereby ensuring the privacy of other files on the device to be projected. When performing screen projection verification, the activity information on the device to be projected is processed to obtain the projection instruction, thereby avoiding the presence of other devices joining the projection content, further improving the security of screen projection.

[0067] 3. In the present invention, when the generated projection content is transmitted to the projection device according to the matching results and primary and secondary information, the content to be displayed on the secondary projection device is cut and transmitted in batches, thereby avoiding the delay caused by the first use of the secondary projection device, and thus improving the smoothness of switching the projection content. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 It is a flowchart of the multi-application window projection management system in the present invention;

[0070] Figure 2 A schematic diagram of the process of generating screen projection content and transmitting it to the screen projection device in the present invention;

[0071] Figure 3 A logic judgment block diagram for processing after obtaining the user's operation instructions on the screen projection device in the present invention;

[0072] Figure 4 It is a connection diagram of the modules in the present invention. DETAILED DESCRIPTION

[0073] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] See also Figure 1 - Figure, the first aspect of the present invention provides a multi-application window projection management system, including: a progressively connected window management module, a projection management module and a touch processing module;

[0075] Window management module: identifying multiple application windows on multiple devices to be projected; managing multiple application windows on the devices to be projected according to operation instructions; and,

[0076] Get the activity information of the current window on the device to be projected and store it in the window list; the activity information includes the window ID, resolution, location and application name of the window;

[0077] Screen projection management module: distinguish the devices to be projected into primary and secondary and generate and send screen projection instructions based on activity information; input the screen projection instructions to the screen projection device and verify them; send the primary and secondary information and window ID of the devices to be projected to the screen projection device according to the verification results; wherein the primary and secondary information includes the primary projection device and the secondary projection device; and,

[0078] Match the window ID obtained by the projection device with the activity information in the window list, and transmit the generated projection content to the projection device according to the matching result and primary and secondary information;

[0079] Touch processing module: obtains the user's operation instructions on the projection device and then processes them; wherein the operation instructions include interaction information and sub-switching commands; and,

[0080] Control the device to be projected or the projected device that has been verified according to the operation instructions.

[0081] For example, there is a computer as the main projection device, and two windows are running, namely office and PDF reader, and the resolution of the two windows is 960×1080, and they are displayed side by side;

[0082] Then obtain the activity information of the current window on the device to be projected, including:

[0083] Get the number of running windows on the device to be projected and mark it as 2; mark the two windows according to the length of their running time to obtain the window ID. Specifically, the window ID of Office is ID1, and the window ID of the PDF reader is ID2;

[0084] The resolutions of the two windows in the device to be projected are counted respectively, and marked as 9601×10801 and 9602×10802;

[0085] Establish an xy plane with the center point of the device to be projected as the origin and pixels as the unit, obtain the coordinates of the upper left corners of the two windows, namely (0,1080) and (960,1080), and mark them as the occupied positions;

[0086] Use the pygetwindow library in Python to extract the application names of several running windows.

[0087] In this embodiment, the devices to be projected are distinguished into primary and secondary devices and a projection instruction is generated and sent based on the activity information, including:

[0088] Mark the windows to be projected on several devices as projection windows, select one device to be projected from the devices to be projected as the primary projection device, and mark the other devices to be projected as secondary projection devices;

[0089] The activity information corresponding to the primary projection device and the secondary projection device are obtained respectively, and several projection instructions are generated based on the corresponding activity information and the type of the device to be projected, and the corresponding projection instructions are sent to the verification end through the corresponding device to be projected; wherein the type of the device to be projected includes the primary projection device and the secondary projection device.

[0090] In order to further improve the security of screen projection, several screen projection instructions are generated based on the corresponding activity information and the type of device to be projected, including:

[0091] Extract the type of the device to be projected, mark the projection window corresponding to the primary projection device in the device to be projected as 0, and mark the projection window corresponding to the secondary projection device in the device to be projected as 1;

[0092] The window ID, resolution, location and application name corresponding to each projection window are extracted and hashed to obtain a hash value; a combination of 4 consecutive letters and numbers is randomly selected from the hash value and marked as a projection instruction.

[0093] The screen projection instructions generated in this way have a certain degree of randomness, and the same screen projection instructions cannot be easily obtained. In addition, the window position and other factors will inevitably be different each time the screen is projected. Therefore, even if the screen projection content is the same, the screen projection instructions will also change. In addition, the instructions of each device to be projected are different, which ensures that the screen projection instructions cannot be easily cracked, and can avoid screen projection instruction leakage, the presence of other interfering devices projecting on the screen projection device, and the inability to reversely obtain the original activity information.

[0094] In order to achieve a smooth match, enter the screen projection command into the projection device and verify it, including:

[0095] Through the notification method, the user enters a 4-digit screen projection command in the screen projection device, and the screen projection device sends the screen projection command to the verification end;

[0096] The verification end determines whether the screen projection instruction sent by the device to be projected is the same as the screen projection instruction sent by the projection device; if yes, the verification result of the device to be projected is marked as passed verification; if not, the verification result of the device to be projected is marked as verification failure, and the projection instruction can be re-entered at this time.

[0097] Furthermore, according to the verification result, the primary and secondary information and the window ID of the device to be projected are sent to the projection device, including:

[0098] Determine whether the verification result of the device to be projected is passed; if yes, extract the primary and secondary information of the device to be projected and the window ID that needs to be projected, and send the primary and secondary information and window ID of the device to be projected to the projection device; if no, input the projection command to the projection device and verify it.

[0099] In order to further reduce the sluggishness caused by switching between the primary and secondary projection devices, and between secondary projection devices, the generated projection content is transmitted to the projection device according to the matching results and the primary and secondary information, including:

[0100] Obtain the content displayed by the matched window ID on the main projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a main projection data packet; transmit the main projection data packet to the projection device;

[0101] Obtain the content displayed by the matched window ID on the secondary projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a secondary projection data packet;

[0102] Extract the file information of the secondary projection data packet and the transmission speed between the secondary projection device and the screen projection device; cut the secondary projection data packet according to the file information and the transmission speed to obtain several secondary projection sub-data packets; arrange the several secondary projection sub-data packets and transmit them to the screen projection device; wherein the file information is the file size of the secondary projection data packet.

[0103] By sending multiple data packets to the projection device in advance for storage in the above manner, processing time can be reduced during subsequent calls.

[0104] Specifically, in order to ensure the speed of the transmitted file and the integrity of the transmitted file, the secondary data packet is cut according to the file information and the transmission speed, including:

[0105] The file size and transmission speed in the file information are extracted and marked as S and V respectively, and the cutting quantile d is calculated by the formula d=V×t / S; where t is the preset transmission time;

[0106] The secondary data packet is cut based on the cutting quantile point to obtain two primary cut data packets;

[0107] The integrity of the two initial data packets is verified; when the two initial data packets are complete, the initial data packets are marked as secondary sub-data packets; when the two initial data packets are incomplete, the two initial sub-data packets are corrected by supplementing and eliminating them respectively to obtain secondary sub-data packets.

[0108] Furthermore, when the second sub-data packet is still large, it can be cut again, and when the two initial cut sub-data packets are corrected, a check field can be added before the data packet is cut, and the data to be supplemented or removed can be determined according to the position of the check field, or an existing network communication protocol framework, such as TCP / IP, UDP, etc., can be used;

[0109] It should be noted that the unit of S is MB, and the unit of V is MB / s. When the sub-data packet is divided into bits, it is ensured that each sub-data contains necessary meta-information so that it can be correctly reassembled at the receiving end. When the meta-information cut according to the cutting point is incomplete, the incomplete information can be selected to be removed or supplemented, so as to ensure the integrity of the content in the subsequent screen projection.

[0110] Specifically, several sub-data packets are arranged and transmitted to the screen projection device, including:

[0111] Based on the sequential marking i of the secondary devices that have passed the verification, the secondary sub-data packets corresponding to the secondary devices are marked as Bij according to the composition order of the data packets; wherein Bij is the jth data packet cut from the secondary data packet in the i-th secondary device; wherein i and j are both positive integers;

[0112] Arrange several sub-projection data packets in a transmission order specified by the transmission rule to obtain a data queue; and transmit several sub-projection data packets to the screen projection device according to the data queue.

[0113] For example, there are two secondary devices that have passed the verification successively, and each secondary device corresponds to two secondary sub-data packets, and the labels are respectively: B11, B12, B21 and B22;

[0114] Furthermore, the transmission order specified by the transmission rules includes:

[0115] Generate a matrix of the labels B11, B12, B21 and B22 corresponding to the sub-data packets of several times Arrange the markers B11, B12, B21 and B22 in the order of the column vectors from left to right, and obtain the marker sequence B11, B21, B12, B22;

[0116] The plurality of sub-data packets corresponding to the marks are sorted according to the mark order to obtain the transmission order.

[0117] In this embodiment, the operation instruction of the user on the screen projection device is obtained and then processed, including:

[0118] Step 1: extracting the interaction information of the interactive device; wherein the interaction information is the indicated coordinates, timestamp, control instructions and moving direction of the indicated point of the interactive device;

[0119] Step 2: Determine whether the interactive information is a basic instruction of the interactive device; if yes, send the interactive information to the corresponding primary projection device or secondary projection device; if no, jump to step 3; wherein the basic instruction is the basic operation of the interactive device;

[0120] Step three: Determine whether there is a match between the interactive information and the screen projection switching instruction library; if so, switch the main projection device and the secondary projection device according to the sub-switching command of the matching result; if not, do not perform any operation.

[0121] Specifically, if the interactive device is a mouse, a camera or a laser pen, the corresponding interactive information is the indicated coordinates, timestamp, control instructions and moving direction generated by the mouse pointer, the camera replenishment gesture or the moving trajectory of the laser pen.

[0122] Furthermore, the sub-switching commands in the screen casting switching instruction library are set based on the interactive information;

[0123] The sub-switching command includes the main-secondary device switching command, the secondary-secondary device switching command, the previous window switching command and the next window switching command.

[0124] The sub-switching commands mentioned above are set based on the interactive information, including:

[0125] Extract the indicated coordinates of the indicated point of the interactive device, and set a preset central rectangle of Chi Xun with the indicated coordinates as the center, and extend the line segments along the four right angles intersecting the line segments from the four right angles of the central rectangle as the starting point to obtain the demarcation line;

[0126] Mark the four areas enclosed by two adjacent demarcation lines and the central rectangular boundary as the operation termination area;

[0127] The operations of moving the pointer coordinates with control instructions from the central rectangle to the four operation termination areas within the preset time are marked as the primary-secondary device switching command, the secondary-secondary device switching command, the previous window switching command and the next window switching command respectively.

[0128] Further, controlling the screen projection device or the screen projection device that has been verified according to the operation instruction includes:

[0129] S1: Determine whether the operation instruction is interactive information; if yes, control the verified device to be projected to which the current display window on the projection device belongs according to the interactive information; if no, jump to S2;

[0130] S2: Determine whether the interactive information is a primary-to-secondary device switching command or a secondary-to-secondary device switching command; if yes, call the primary data packet or secondary data packet corresponding to the verified device to be projected through the projection device, and display it on the projection device; if no, control the verified device to be projected to which the current display window on the projection device belongs to perform the corresponding window switching.

[0131] It should be noted that the verified devices to be projected include the primary projection device and the secondary projection device.

[0132] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-application window projection management system as described in the first aspect above.

[0133] A computer program product is used to run on a computer and / or a terminal device to implement a multi-application window projection management system as described in the first aspect above.

[0134] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0135] The working principle of the present invention is as follows: first, by identifying multiple application windows on multiple devices to be projected, the activity information of the current window on the device to be projected is obtained and stored in a window list; the devices to be projected are distinguished into primary and secondary, and a projection instruction is generated and sent based on the activity information, and the user inputs the projection instruction to the projection device and verifies it; according to the verification result, the primary and secondary information and window ID of the device to be projected are sent to the projection device, the window ID obtained by the projection device is matched with the activity information in the window list, and the generated projection content is transmitted to the projection device according to the matching result and the primary and secondary information; when in use, the user's operation instructions on the projection device are obtained and then processed, and the device to be projected or the projection device that has been verified according to the operation instructions is controlled, and multiple application windows on the device to be projected are managed according to the operation instructions.

[0136] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-application window projection management system, comprising: Window management module, screen projection management module and touch processing module; characterized in that, The window management module: identifies multiple application windows on multiple devices to be projected; manages the multiple application windows on the devices to be projected according to the operation instructions; and Get the activity information of the current window on the device to be projected and store it in the window list; the activity information includes the window ID, resolution, location and application name of the window; The screen projection management module: distinguishes the devices to be projected into primary and secondary and generates and sends a screen projection instruction based on the activity information; inputs the screen projection instruction to the screen projection device and verifies it; sends the primary and secondary information and window ID of the devices to be projected to the screen projection device according to the verification result; wherein the primary and secondary information includes the primary projection device and the secondary projection device; and, Match the window ID obtained by the projection device with the activity information in the window list, and transmit the generated projection content to the projection device according to the matching result and primary and secondary information; The touch control processing module: processes the operation instructions of the user on the screen projection device after obtaining the operation instructions; wherein the operation instructions include interaction information and sub-switching commands; and, Control the device to be projected or the projected device that has been verified according to the operation instructions.

2. A multi-application window projection management system according to claim 1, characterized in that: The method of obtaining the activity information of the current window on the device to be projected includes: Get the number of windows running on the device to be projected and mark it as n; mark the windows according to the running time of each of the n windows to obtain the window ID; Count the resolutions occupied by n windows in the device to be projected, and mark them as xk×yk; where n is a positive integer, x represents the number of pixels occupied by the window horizontally, y represents the number of pixels occupied vertically, and k∈[1,n]; Establish an xy plane with the center point of the device to be projected as the origin and pixels as the unit, obtain the coordinates of the upper left corners of several windows, and mark them as the occupied positions; Use Python program to extract the application names of several running windows.

3. A multi-application window projection management system according to claim 1, characterized in that: The step of distinguishing the primary and secondary devices to be projected and generating and sending a projection instruction based on the activity information includes: Mark the windows to be projected on several devices as projection windows, select one device to be projected from the devices to be projected as the primary projection device, and mark the other devices to be projected as secondary projection devices; The activity information corresponding to the primary projection device and the secondary projection device are obtained respectively, and several projection instructions are generated based on the corresponding activity information and the type of the device to be projected, and the corresponding projection instructions are sent to the verification end through the corresponding device to be projected; wherein the type of the device to be projected includes the primary projection device and the secondary projection device.

4. A multi-application window projection management system according to claim 3, characterized in that: The generating of several screen projection instructions based on the corresponding activity information and the type of the device to be projected includes: Extract the type of the device to be projected, mark the projection window corresponding to the primary projection device in the device to be projected with features, and mark the projection window corresponding to the secondary projection device in the device to be projected with non-features; The window ID, resolution, location and application name corresponding to each projection window are extracted and hashed to obtain a hash value; a combination of n consecutive letters and numbers is randomly selected from the hash value and marked as a projection instruction.

5. A multi-application window projection management system according to claim 1, characterized in that: The step of inputting a screen projection instruction to the screen projection device and verifying the instruction includes: Enter the screen projection command in the screen projection device, and the screen projection device sends the screen projection command to the verification end; The verification end determines whether the screen projection instruction sent by the device to be projected is the same as the screen projection instruction sent by the projection device; if yes, the verification result of the device to be projected is marked as passed verification; if not, the verification result of the device to be projected is marked as verification failure.

6. A multi-application window projection management system according to claim 1, characterized in that: The step of sending the primary and secondary information and the window ID of the device to be projected to the projection device according to the verification result includes: Determine whether the verification result of the device to be projected is passed; if yes, extract the primary and secondary information of the device to be projected and the window ID that needs to be projected, and send the primary and secondary information and window ID of the device to be projected to the projection device; if no, input the projection command to the projection device and verify it.

7. A multi-application window projection management system according to claim 1, characterized in that: The generating screen projection content and transmitting it to the screen projection device according to the matching result and the primary and secondary information includes: Obtain the content displayed by the matched window ID on the main projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a main projection data packet; transmit the main projection data packet to the projection device; Obtain the content displayed by the matched window ID on the secondary projection device, and capture the content displayed on the window ID through screen capture technology and then encode and compress it to obtain a secondary projection data packet; Extract the file information of the secondary projection data packet and the transmission speed between the secondary projection device and the screen projection device; cut the secondary projection data packet according to the file information and the transmission speed to obtain several secondary projection sub-data packets; arrange the several secondary projection sub-data packets and transmit them to the screen projection device; wherein the file information is the file size of the secondary projection data packet.

8. A multi-application window projection management system according to claim 7, characterized in that: The cutting of the secondary data packet according to the file information and the transmission speed includes: The file size and transmission speed in the file information are extracted and marked as S and V respectively, and the cutting quantile d is calculated by the formula d=V×t / S; where t is the preset transmission time; The secondary data packet is cut based on the cutting quantile point to obtain two primary cut data packets; The integrity of the two initial data packets is verified; when the two initial data packets are complete, the initial data packets are marked as secondary sub-data packets; when the two initial data packets are incomplete, the two initial sub-data packets are corrected by supplementing and eliminating them respectively to obtain secondary sub-data packets.

9. A multi-application window projection management system according to claim 7, characterized in that: The method of arranging the plurality of projection sub-data packets and transmitting them to the projection device includes: Based on the sequential marking i of the secondary devices that have passed the verification, the secondary sub-data packets corresponding to the secondary devices are marked as Bij according to the composition order of the data packets; wherein Bij is the jth data packet cut from the secondary data packet in the i-th secondary device; wherein i and j are both positive integers; Arrange several sub-projection data packets in a transmission order specified by the transmission rule to obtain a data queue; and transmit several sub-projection data packets to the screen projection device according to the data queue.

10. A multi-application window projection management system according to claim 9, characterized in that: The transmission order specified by the transmission rule includes: Generate a matrix [Bij] with the labels Bij corresponding to the sub-data packets of the multiple times, and arrange the labels Bij in the order of the column vectors from left to right to obtain the label order; The plurality of sub-data packets corresponding to the marks are sorted according to the mark order to obtain the transmission order.

11. A multi-application window projection management system according to claim 1, characterized in that: The processing after obtaining the operation instruction of the user on the screen projection device includes: Step 1: extracting interaction information of the interaction device; wherein the interaction information is the indicated coordinates, timestamp, control instructions and moving direction of the indicated point of the interaction device; Step 2: Determine whether the interactive information is a basic instruction of the interactive device; if yes, send the interactive information to the corresponding primary projection device or secondary projection device; if no, jump to step 3; wherein the basic instruction is the basic operation of the interactive device; Step three: Determine whether there is a match result when matching the interactive information with the screen projection switching instruction library; if so, switch the main projection device and the secondary projection device according to the sub-switching command of the matching result; if not, do not perform any operation.

12. A multi-application window projection management system according to claim 11, characterized in that: The sub-switching commands in the screen projection switching instruction library are set based on the interaction information; The sub-switching command includes a primary-secondary device switching command, a secondary-secondary device switching command, a previous window switching command and a next window switching command.

13. A multi-application window projection management system according to claim 12, characterized in that: The sub-switching command is set based on the interactive information, including: Extract the indicated coordinates of the indicated point of the interactive device, and set a preset central rectangle of Chi Xun with the indicated coordinates as the center, and extend the line segments along the four right angles intersecting the line segments from the four right angles of the central rectangle as the starting points to obtain the demarcation line; Mark the four areas enclosed by two adjacent demarcation lines and the central rectangular boundary as the operation termination area; The operations of moving the pointer coordinates with control instructions from the central rectangle to the four operation termination areas within the preset time are marked as the primary-secondary device switching command, the secondary-secondary device switching command, the previous window switching command and the next window switching command respectively.

14. A multi-application window projection management system according to claim 1, characterized in that: The controlling of the screen projection device or the screen projection device that has been verified according to the operation instruction includes: S1: Determine whether the operation instruction is interactive information; if yes, control the verified device to be projected to which the current display window on the projection device belongs according to the interactive information; if no, jump to S2; S2: Determine whether the interactive information is a primary-to-secondary device switching command or a secondary-to-secondary device switching command; if yes, call the primary data packet or secondary data packet corresponding to the verified device to be projected through the projection device, and display it on the projection device; if no, control the verified device to be projected to which the current display window on the projection device belongs to perform the corresponding window switching.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multi-application window projection management system as described in any one of claims 1 to 14.

16. A computer program product, characterized in that The computer program product is used to run on a computer and / or terminal device to implement the multi-application window projection management system as described in claims 1 to 14.