Image data processing method, zero terminal, computer equipment and storage medium

Abstract

The invention provides an image data processing method, a zero terminal, computer equipment and a storage medium. The image data processing method comprises the following steps: receiving a main screen code stream and an auxiliary screen code stream sent by a server, and respectively decoding the main screen code stream and the auxiliary screen code stream to generate main screen decoding data and auxiliary screen decoding data; determining a screen isolation function level selected by the user; wherein the screen isolation function level comprises a forcing level and a warning level; according to the main screen decoding data and the auxiliary screen decoding data, whether a user drags a current display window of the auxiliary screen to the main screen or not is judged; and if the user drags the current display window of the auxiliary screen to the main screen, performing corresponding operation according to the screen isolation function level. According to the image data processing method disclosed by the invention, the window content on the secondary screen is forbidden to be dragged to the primary screen in a double-screen (extended screen) mode, so that the possible risk of information leakage is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image data processing method and a terminal, computer equipment, and storage medium. Background Technology

[0002] The term "zero-terminal" specifically refers to a terminal box, which exists as the client of the overall system. The overall system consists of two main parts: a server and a client. The server is a physical machine or virtual machine operating system running modules for image acquisition and encoding. It can acquire, encode, compress, and transmit the raw screen display content over the network to the terminal box. The terminal box decodes the bitstream and displays it on the screen through display ports such as HDMI, allowing the end user to remotely use the server's computer. This disclosure uses a dual-screen scenario, where the server-side computer is set to dual-screen (extended screen) mode, and two screens are simultaneously inserted into the zero-terminal. For the user, this is equivalent to a dual-screen office scenario, and the user's final experience is the same as using a dual-screen (extended screen) office on a local computer. Due to the inherent characteristics of dual screens, users can drag windows from one screen to another, and any window can be displayed on either screen. However, this feature has limitations in certain scenarios. In some situations, it is required that the two screens display different themes. For example, in some factory or subway scenarios, the main screen is required to display real-time monitoring or simulated operation diagrams, while the secondary screen is used for daily office operations by staff. Even if the two screens are physically separated, this can still be a significant difference. If staff members accidentally drag and drop their daily work content onto the main screen's simulated operation graph, unrelated personnel may see it, potentially leading to information leaks or other unnecessary problems. Furthermore, since dual-screen functionality is an inherent feature of operating systems, current zero-terminal product solutions do not specifically address this issue, posing certain security risks. Summary of the Invention

[0003] The purpose of this disclosure is to overcome the shortcomings of the prior art and provide an image data processing method, a terminal, a computer device, and a storage medium. This image data processing method can prevent the content of a window on the secondary screen from being dragged to the main screen in dual-screen (extended screen) mode, so as to avoid the risk of potential information leakage.

[0004] According to a first aspect of the present disclosure, an image data processing method is provided, applied to a zero-terminal device, the method comprising: The system receives the main screen bitstream and the secondary screen bitstream sent by the server, and decodes the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; Determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; Based on the main screen decoding data and the secondary screen decoding data, determine whether the user has dragged the current display window of the secondary screen onto the main screen; If the user drags the currently displayed window of the secondary screen onto the main screen, the corresponding operation will be performed according to the screen isolation function level.

[0005] In one embodiment, performing the corresponding operation based on the screen isolation function level includes: If the user's level is mandatory, then pause responding to the user's mouse drag events; Draw lines around the outer edge of the current display window on the secondary screen to prompt the user to drag the current display window across screens at the forced level.

[0006] In one embodiment, performing the corresponding operation based on the screen isolation function level includes: If the user's level is warning level, analyze and calculate the position and size of the secondary screen display window dragged into the main screen; Based on the position and size of the secondary screen display window dragged into the main screen, fill the corresponding area of ​​the main screen with a solid color to achieve information masking.

[0007] In one embodiment, determining whether the user drags the current display window of the secondary screen onto the main screen based on the main screen decoding data and the secondary screen decoding data includes: Get the position and size of the currently displayed window on the secondary screen in the current frame; Get the position and size of the current display window of the secondary screen in the previous frame; The distance from the left edge of the sub-screen to the left edge of the sub-screen is calculated based on the difference between the position and size of the current display window of the sub-screen in the current frame and the position and size of the current display window of the sub-screen in the previous frame. When a mouse drag event is detected again, calculate the offset between the target position of the horizontal mouse drag and the current mouse position; If the offset between the target position of the horizontal mouse drag and the current mouse position is greater than the distance between the left edge of the current display window of the secondary screen and the left edge of the secondary screen, then it is determined that the user has dragged the current display window of the secondary screen onto the main screen.

[0008] In one embodiment, analyzing and calculating the position and size of the secondary screen display window dragged into the main screen portion includes: Get the position and size of the currently displayed window on the secondary screen in the current frame; Get the position and size of the current display window of the secondary screen in the previous frame; The motion vector of the current display window of the secondary screen is calculated based on the position and size of the current display window in the current frame and the position and size of the current display window of the secondary screen in the previous frame. Calculate the target position and size of the current display window of the secondary screen in the next frame based on the motion vector of the current display window of the secondary screen; Obtain the actual position and size of the current display window of the secondary screen in the next frame after the current frame; Calculate the difference between the target position and size of the current display window of the secondary screen in the next frame and the actual position and size of the current display window of the secondary screen in the next frame; The position and size of the secondary screen display window dragged into the main screen are calculated based on the difference.

[0009] In one embodiment, determining the screen isolation feature level selected by the user includes: Retrieve keyboard and mouse events that indicate user actions controlling the on / off state of screen isolation. The screen isolation level selected by the user is determined based on the keyboard and mouse events that control the on / off switch of the screen isolation function.

[0010] According to a second aspect of the present disclosure, a zero-terminal is provided, the zero-terminal comprising: a processing module, a determining module, a judging module, and an operation module; wherein, The processing module is used to receive the main screen bitstream and the secondary screen bitstream sent by the server, and decode the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; The determining module is used to determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; The judgment module is used to determine whether the user has dragged the current display window of the secondary screen onto the main screen based on the main screen decoding data and the secondary screen decoding data; The operation module is used to perform corresponding operations based on the screen isolation function level when the user drags the current display window of the secondary screen onto the main screen.

[0011] In one embodiment, the zero terminal further includes: a first decoder, a second decoder, a level control switch, a mouse event limiting module, a reconstruction buffer connected to both the first and second decoders, a main screen image modification module connected to the reconstruction buffer, a second display module, an inappropriate content area calculation module, and a first display module connected to the main screen image modification module; wherein, The first decoder is used to decode the main screen bitstream and generate main screen decoded data; The second decoder is used to decode the secondary screen bitstream and generate secondary screen decoded data; The reconstruction buffer is used to temporarily store the main screen decoding data and the secondary screen decoding data; The first display module is used to send the decoded data of the main screen to the main screen for display; The main screen image modification module is used to modify the area displaying the current display window content of the secondary screen on the main screen to a solid color when the screen isolation function level is warning level, so as to send the main screen decoded data to the main screen for display after information is blocked; The second display module is used to send the decoded data from the secondary screen to the secondary screen for display. The inappropriate content area calculation module is used to calculate the position and size of the secondary screen display window dragged into the main screen when the screen isolation function level is warning level, and submit the calculation result to the main screen image modification module; The level control switch is used by the user to set the forced level, warning level, and disable the screen isolation function. The mouse event limiting module is used to pause the current mouse dragging event when the screen isolation function level is at the warning level and it is determined that the user drags the current display window of the secondary screen onto the main screen.

[0012] According to a third aspect of the present disclosure, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0013] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method as described in any of the above.

[0014] This disclosure provides an image data processing method that optimizes zero-terminal products with extended screens. In extended dual-screen mode, it uses two levels to control user window dragging. If dragging occurs from one screen to another, it promptly disables the window and issues an alarm, ensuring the information security of a specific screen. This technology is practical in situations with high information security requirements. Furthermore, this disclosure places no requirements on the server-side operating system and software; all functions are implemented on the zero-terminal device, thereby expanding the application scope of this disclosure. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0016] Figure 1 This is a schematic diagram of a zero-terminal system provided in an embodiment of this disclosure.

[0017] Figure 2 This is a schematic diagram illustrating a scenario in the prior art disclosed herein where inappropriate content appears on the screen.

[0018] Figure 3 This is a schematic diagram of mouse event control at the forced level in an image data processing method provided in an embodiment of this disclosure.

[0019] Figure 4 This is a schematic diagram of a method for identifying inappropriate content regions in an image data processing method provided in this embodiment of the present disclosure.

[0020] Figure 5 This is a flowchart illustrating an image data processing method provided in an embodiment of the present disclosure.

[0021] Figure 6 This is a flowchart illustrating an image data processing method provided in an embodiment of the present disclosure.

[0022] Figure 7 This is a flowchart illustrating an image data processing method provided in an embodiment of the present disclosure.

[0023] Figure 8 This is a diagram illustrating a zero-terminal architecture provided in an embodiment of this disclosure.

[0024] Figure 9 This is a diagram illustrating a zero-terminal architecture provided in an embodiment of this disclosure.

[0025] Figure 10 This is an architecture diagram of a server provided in an embodiment of the present disclosure.

[0026] Figure 11 This is an architectural diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] Figure 1 This is a schematic diagram of a zero-terminal system provided as an embodiment of this disclosure. Figure 1As shown, the system consists of two parts: a server and a client, which are connected via a network. Users can simultaneously connect to two desktop images on their terminal, corresponding to the main screen and secondary screen of the remote office machine, respectively. These screens operate in extended screen mode. The remote office machine runs a screen capture program and an image encoding program, which captures and encodes the desktop screen on the office machine and transmits it to the terminal via the network, i.e., to the zero terminal.

[0029] A decoding program runs on the zero terminal. After decoding, the zero terminal displays the reconstructed image on two monitors. This is how the zero terminal works. The two monitors connected to the terminal correspond to the main screen and secondary screen of the server computer, respectively. In the current solution, the user can drag and drop the window content between the two screens.

[0030] Figure 2 This is a schematic diagram illustrating a scenario where inappropriate content appears on a screen, as described in the prior art of this disclosure. The problem this disclosure aims to solve can be addressed using... Figure 2 To indicate, in Figure 2 In this scenario, the main screen displays a real-time animated map of underground transportation operations; its primary purpose is public display, allowing anyone to view it. The secondary screen, on the other hand, is used by office staff for daily tasks, such as... Figure 2 As shown, an office worker inadvertently dragged a confidential document window onto the main screen. Since the main and secondary screens may not be in the same physical location, if the office worker does not notice this content on the main screen, but an external person viewing the main screen does, it could lead to a leak. This content, which should not be viewed by external personnel, is referred to in this disclosure as "inappropriate content." This disclosure is made to prevent the leakage of inappropriate content.

[0031] The core technology for addressing inappropriate content areas is identifying which parts constitute inappropriate regions. Since this disclosure has no dependency on the server-side computer system, all detection work must be completed on the client side. Furthermore, the zero requirement for the server-side system is one of the advantages of this disclosure. In this disclosure, the detection of inappropriate regions on the client side is calculated and manipulated at the pixel level. That is, the terminal device does not need to first decode the raw bitstream received from the network into YUV data before performing analysis and processing on the YUV data.

[0032] It should be noted that the scenarios described in this disclosure assume that the main screen is used as a publicly accessible screen and the secondary screen is used as an employee's work screen. In practice, there are also cases where the main screen is used as an office screen and the secondary screen is used as a publicly accessible screen. In such cases, it is only necessary to swap the main and secondary modules in this disclosure. The specific implementation method is the same, so it will not be described separately.

[0033] The zero-terminal disclosed herein supports two levels of screen isolation: a mandatory level and a warning level. Specifically, the zero-terminal interface includes a three-level switch to control the screen isolation function, with the three levels being "mandatory / warning / off". Users can specify which level to use. Setting it to "off" indicates that the screen isolation function is not used, and the window can be dragged freely.

[0034] The following describes the core method for implementing screen isolation functionality on the zero-terminal when set to the mandatory level. It's called the mandatory level because in this case, the zero-terminal determines whether the user is dragging a window from the secondary screen to the primary screen at the pixel level. Once it detects this, it pauses mouse drag events and draws a dashed line around the edge of the dragged window to indicate that the cross-screen dragging is happening under the mandatory security level. At this level, the original view of the primary screen remains unaffected, and the window dragging action is terminated by disabling mouse drag events at the moment of screen crossing. The key point is how to determine when a cross-screen window dragging is occurring. Figure 3 This illustrates the situation.

[0035] like Figure 3 As shown, in frame 1, the window is at position 1. In frame 2, the window is dragged to position 2. During this dragging process, the motion vectors (or trajectories) of all image-coded blocks within the window are consistent. Based on this characteristic, the zero-terminal system calculates the window's position and size in frame 2 and analyzes the distance between the window's left edge and the left edge of the sub-screen, representing this distance as a margin variable. When the next mouse drag event occurs, the system analyzes the offset between the mouse's horizontal drag target position and its current position. If the horizontal span of this mouse event exceeds the current margin value, meaning the window will be dragged outside the left side of the sub-screen, this mouse drag event is deemed disallowed and prohibited. A dashed frame is then overlaid on the outer edge of the window at position 2 to warn the user that the window's position is now forcibly restricted to within the sub-screen's boundaries. It should be noted that this dashed frame is drawn on the decoded YUV image of the current sub-screen. This level offers the highest security, but users will experience a relatively strong sense of restriction during operation.

[0036] The following describes the core method for implementing screen isolation functionality in the zero-terminal when set to the warning level. At the warning level, mouse dragging is not restricted. When a user drags a window out of the secondary screen and into the main screen, the zero-terminal analyzes and calculates the position and size of the portion of the window dragged into the main screen, filling this area with a solid color to achieve information masking. Compared to the mandatory level, the warning level does not interfere with the user's mouse operations, offering less of a sense of restriction, but it displays a solid-color area on the main screen—the inappropriate information area. The core implementation of the warning level primarily involves analyzing and determining the position and size of the inappropriate area spanning the screen. Figure 4 The determination method is illustrated.

[0037] exist Figure 4 In this scenario, assuming there are three adjacent frames controlled by mouse dragging, a document window is dragged from the upper right corner of the secondary screen to the far left, and the left half of the document has already appeared in the main screen area. This area that was already on the main screen is the inappropriate content area we need to identify. The method for determining this area is as follows: On the secondary screen, when the window moves from position 1 to position 2, all coded blocks within that window will exhibit the same motion vector, which is: Figure 4 The large arrows at positions 1 and 2 indicate the content of the window. All coded blocks that match this motion vector represent the content of this window. The motion vector is contained within the bitstream. By counting the macroblocks that match this motion vector, we can analyze the position and size of window 2 in frame 2 across the entire frame's pixel range. Similarly, we can obtain the content of the right half of the actual display area at position 3 in frame 3. Since the dashed area no longer belongs to the secondary screen, it is not included in the secondary screen's bitstream. Therefore, by performing a simple subtraction calculation based on the window position and size at position 2 and the right half of the window position and size at position 3, we can obtain the position and size of the dashed area, which is the information about the inappropriate content area on the main screen.

[0038] After obtaining the location and size of the inappropriate content area on the main screen, the zero terminal modifies the YUV information in that area to a certain monochrome. Optionally, it can also render text prompts such as "Confidential Area" in that area as a notification to external personnel or the employee.

[0039] Figure 5 This is a schematic flowchart illustrating an image data processing method provided in an embodiment of this disclosure. Figure 5 As shown, this method is applied to zero terminals and includes: Step 501: Receive the main screen bitstream and the secondary screen bitstream sent by the server, and decode the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; In this step, the first decoder in the zero terminal decodes the main screen bitstream acquired by the main screen image acquisition module and sent through the first transmission channel, and the second decoder decodes the secondary screen bitstream acquired by the secondary screen image acquisition module and sent through the second transmission channel, thereby generating main screen decoding data and secondary screen decoding data respectively.

[0040] Step 502: Determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; In one embodiment, determining the screen isolation feature level selected by the user includes: Retrieve keyboard and mouse events that indicate user actions controlling the on / off state of screen isolation. The screen isolation level selected by the user is determined based on the keyboard and mouse events that control the on / off switch of the screen isolation function.

[0041] In this embodiment, the zero terminal determines the screen isolation function level selected by the user by acquiring the on / off mouse event of the user's operation to control the screen isolation function level. The switch is set with a forced level, a warning level, and a screen isolation function off.

[0042] Step 503: Based on the main screen decoding data and the secondary screen decoding data, determine whether the user drags the current display window of the secondary screen onto the main screen; In one embodiment, such as Figure 6 As shown, determining whether the user drags the current display window of the secondary screen onto the main screen based on the main screen decoding data and the secondary screen decoding data includes: Step 601: Obtain the position and size of the currently displayed window of the secondary screen in the current frame; Step 602: Obtain the position and size of the current display window of the secondary screen in the previous frame; Step 603: Calculate the distance from the left edge of the sub-screen to the left edge of the sub-screen based on the difference between the position and size of the current display window of the sub-screen in the current frame and the position and size of the current display window of the sub-screen in the previous frame; Step 604: When a mouse drag event is detected again, calculate the offset between the target position of the horizontal mouse drag and the current mouse position; Step 605: If the offset between the target position of the horizontal mouse drag and the current mouse position is greater than the distance between the left edge of the current display window of the secondary screen and the left edge of the secondary screen, then it is determined that the user has dragged the current display window of the secondary screen onto the main screen.

[0043] In this embodiment, the selected screen isolation level is the mandatory level. Because the motion vectors (or trajectories) of all image-encoded blocks within the window are consistent during dragging, the zero-terminal system calculates the window's position and size in the second frame and analyzes the distance between the window's left edge and the left edge of the secondary screen, representing this distance as a margin variable. When the next mouse drag event occurs, the system analyzes the offset between the mouse's horizontal drag target position and its current position. If the horizontal span of this mouse event exceeds the current margin value—meaning the window will be dragged outside the left side of the secondary screen—it is determined that the mouse drag event is not allowed and will be prohibited. A dashed frame will be overlaid on the outer edge of the window at position 2 to warn the user that the window's position is now forcibly restricted within the secondary screen's range. It should be noted that this dashed frame is drawn on the decoded YUV image of the current secondary screen. This level offers the highest security, but users will experience a relatively strong sense of restriction during operation.

[0044] Step 504: If the user drags the current display window of the secondary screen onto the main screen, then perform the corresponding operation according to the screen isolation function level.

[0045] In one implementation, the step of performing the corresponding operation based on the screen isolation function level includes: If the user's level is mandatory, then pause responding to the user's mouse drag events; Draw lines around the outer edge of the current display window on the secondary screen to prompt the user to drag the current display window across screens at the forced level.

[0046] In another implementation, the corresponding operation based on the screen isolation function level includes: If the user's level is warning level, analyze and calculate the position and size of the secondary screen display window dragged into the main screen; In one embodiment, such as Figure 7 As shown, the analysis and calculation of the position and size of the secondary screen display window dragged into the main screen includes: Step 701: Obtain the position and size of the currently displayed window of the secondary screen in the current frame; Step 702: Obtain the position and size of the current display window of the secondary screen in the previous frame; Step 703: Calculate the motion vector of the current display window of the secondary screen based on its position and size in the current frame and its position and size in the previous frame. Step 704: Calculate the target position and size of the current display window of the secondary screen in the next frame based on the motion vector of the current display window of the secondary screen; Step 705: Obtain the actual position and size of the current display window of the secondary screen in the next frame after the current frame; Step 706: Calculate the difference between the target position and size of the current display window of the secondary screen in the next frame and the actual position and size of the current display window of the secondary screen in the next frame; Step 707: Calculate the position and size of the secondary screen display window dragged into the main screen based on the difference.

[0047] Based on the position and size of the secondary screen display window dragged into the main screen, fill the corresponding area of ​​the main screen with a solid color to achieve information masking.

[0048] In this implementation, on the secondary screen, when the window moves from position 1 to position 2, all coded blocks within that window will exhibit the same motion vector, which is: Figure 4 The large arrows at positions 1 and 2 indicate the content of the window. All coded blocks that match this motion vector represent the content of this window. The motion vector is contained within the bitstream. By counting the macroblocks that match this motion vector, we can analyze the position and size of window 2 in frame 2 across the entire frame's pixel range. Similarly, we can obtain the content of the right half of the actual display area at position 3 in frame 3. Since the dashed area no longer belongs to the secondary screen, it is not included in the secondary screen's bitstream. Therefore, by performing a simple subtraction calculation based on the window position and size at position 2 and the right half of the window position and size at position 3, we can obtain the position and size of the dashed area, which is the information about the inappropriate content area on the main screen.

[0049] After obtaining the location and size of the inappropriate content area on the main screen, the zero terminal modifies the YUV information in that area to a certain monochrome. Optionally, it can also render text prompts such as "Confidential Area" in that area as a notification to external personnel or the employee.

[0050] This disclosure provides an image data processing method that optimizes zero-terminal products with extended screens. In extended dual-screen mode, it uses two levels to control user window dragging. If dragging occurs from one screen to another, it promptly disables the window and issues an alarm, ensuring the information security of a specific screen. This technology is practical in situations with high information security requirements. Furthermore, this disclosure places no requirements on the server-side operating system and software; all functions are implemented on the zero-terminal device, thereby expanding the application scope of this disclosure.

[0051] Figure 8 This is a diagram illustrating a zero-terminal architecture provided in an embodiment of this disclosure. Figure 8 As shown, the zero-terminal includes: a processing module 801, a determining module 802, a judging module 803, and an operation module 804; wherein, the processing module 801 is used to receive the main screen bitstream and the secondary screen bitstream sent by the server, and decode the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; the determining module 802 is used to determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; the judging module 803 is used to determine whether the user drags the current display window of the secondary screen to the main screen based on the main screen decoded data and the secondary screen decoded data; the operation module 804 is used to perform corresponding operations according to the screen isolation function level when the user drags the current display window of the secondary screen to the main screen.

[0052] Figure 9 This is a diagram illustrating a zero-terminal architecture provided in an embodiment of this disclosure. Figure 9 As shown, the zero terminal also includes: a first decoder 901, a second decoder 902, a level control switch 903, a mouse event limiting module 904, a reconstruction buffer 905 connected to both the first decoder 901 and the second decoder 902, a main screen image modification module 906 connected to the reconstruction buffer 905, a second display module 907, an inappropriate content area calculation module 908, and a first display module 909 connected to the main screen image modification module 906; wherein, the first decoder 901 is used to decode the main screen bitstream to generate main screen decoded data; the second decoder 902 is used to decode the secondary screen bitstream to generate secondary screen decoded data; the reconstruction buffer 903 is used to temporarily store the main screen decoded data and the secondary screen decoded data; the first display module 907 is used to send the main screen decoded data to the main screen for display. The main screen image modification module 906 is used to modify the area displaying the content of the current display window of the secondary screen on the main screen to a solid color when the screen isolation function level is warning level, so as to send the decoded data of the main screen to the main screen for display after information is blocked; the second display module 907 is used to display the decoded data of the secondary screen sent to the secondary screen for display; the inappropriate content area calculation module 908 is used to calculate the position and size of the secondary screen display window dragged into the main screen when the screen isolation function level is warning level, and submit the calculation result to the main screen image modification module; the level control switch is used by the user to set the forced level, warning level and turn off the screen isolation function; the mouse event restriction module 903 is used to pause the current mouse dragging event when the screen isolation function level is warning level and it is determined that the user drags the current display window of the secondary screen onto the main screen.

[0053] Figure 10 This is an architecture diagram of a server provided as an embodiment of this disclosure. (For example...) Figure 10 As shown, the server includes: a main screen image acquisition module 1001, a secondary screen image acquisition module 1002, a first encoder 1003 connected to the main screen image acquisition module 1001, a second encoder 1004 connected to the secondary screen image acquisition module 1002, a first transmission channel 1005 connected to the first encoder 1003, and a second transmission channel 1006 connected to the second encoder 1004; wherein, the main screen image acquisition module 1001 is used to acquire the desktop content of the remote computer's main screen in YUV format; the secondary screen image acquisition module 1002 is used to encode the main screen image, generally an H.264 or HEVC encoder; the first transmission channel 1005 is used to transmit the binary stream encoded by the first encoder 1003 to the network, and the first transmission channel 1006 is used to transmit the binary stream encoded by the second encoder 1004 to the network.

[0054] In one embodiment, a computer device is provided, the internal structure of which can be shown in the following diagram. Figure 11 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the image data processing method described above. It includes: memory and a processor; the memory stores a computer program; and the processor executes the computer program to implement any step of the image data processing method described above.

[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can perform any of the steps in the image data processing implementation method described above.

[0056] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0057] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An image data processing method, characterized in that... Applied to zero terminals, the method includes: The system receives the main screen bitstream and the secondary screen bitstream sent by the server, and decodes the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; Determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; Based on the main screen decoding data and the secondary screen decoding data, determine whether the user has dragged the current display window of the secondary screen onto the main screen; If the user drags the currently displayed window of the secondary screen onto the main screen, the corresponding operation will be performed according to the screen isolation function level.

2. The method according to claim 1, characterized in that... The operation based on the screen isolation function level includes: If the user's level is mandatory, then pause responding to the user's mouse drag events; Draw lines around the outer edge of the current display window on the secondary screen to prompt the user to drag the current display window across screens at the forced level.

3. The method according to claim 2, characterized in that... The operation based on the screen isolation function level includes: If the user's level is warning level, analyze and calculate the position and size of the secondary screen display window dragged into the main screen; Based on the position and size of the secondary screen display window dragged into the main screen, fill the corresponding area of ​​the main screen with a solid color to achieve information masking.

4. The method according to claim 2, characterized in that... The step of determining whether the user drags the current display window of the secondary screen onto the main screen based on the main screen decoding data and the secondary screen decoding data includes: Get the position and size of the currently displayed window on the secondary screen in the current frame; Get the position and size of the current display window of the secondary screen in the previous frame; The distance from the left edge of the sub-screen to the left edge of the sub-screen is calculated based on the difference between the position and size of the current display window of the sub-screen in the current frame and the position and size of the current display window of the sub-screen in the previous frame. When a mouse drag event is detected again, calculate the offset between the target position of the horizontal mouse drag and the current mouse position; If the offset between the target position of the horizontal mouse drag and the current mouse position is greater than the distance between the left edge of the current display window of the secondary screen and the left edge of the secondary screen, then it is determined that the user has dragged the current display window of the secondary screen onto the main screen.

5. The method according to claim 3, characterized in that... The analysis and calculation of the position and size of the secondary screen display window dragged into the main screen includes: Get the position and size of the currently displayed window on the secondary screen in the current frame; Get the position and size of the current display window of the secondary screen in the previous frame; The motion vector of the current display window of the secondary screen is calculated based on the position and size of the current display window in the current frame and the position and size of the current display window of the secondary screen in the previous frame. Calculate the target position and size of the current display window of the secondary screen in the next frame based on the motion vector of the current display window of the secondary screen; Obtain the actual position and size of the current display window of the secondary screen in the next frame after the current frame; Calculate the difference between the target position and size of the current display window of the secondary screen in the next frame and the actual position and size of the current display window of the secondary screen in the next frame; The position and size of the secondary screen display window dragged into the main screen are calculated based on the difference.

6. The method according to claim 1, characterized in that... The process of determining the screen isolation function level selected by the user includes: Retrieve keyboard and mouse events that indicate user actions controlling the on / off state of screen isolation. The screen isolation level selected by the user is determined based on the keyboard and mouse events that control the on / off switch of the screen isolation function.

7. A zero-terminal, characterized in that... The zero terminal includes: a processing module, a determining module, a judging module, and an operation module; wherein, The processing module is used to receive the main screen bitstream and the secondary screen bitstream sent by the server, and decode the main screen bitstream and the secondary screen bitstream respectively to generate main screen decoded data and secondary screen decoded data; The determining module is used to determine the screen isolation function level selected by the user; wherein, the screen isolation function level includes a mandatory level and a warning level; The judgment module is used to determine whether the user has dragged the current display window of the secondary screen onto the main screen based on the main screen decoding data and the secondary screen decoding data; The operation module is used to perform corresponding operations based on the screen isolation function level when the user drags the current display window of the secondary screen onto the main screen.

8. The zero-terminal according to claim 7, characterized in that... The zero terminal further includes: a first decoder, a second decoder, a level control switch, a mouse event limiting module, a reconstruction buffer connected to both the first and second decoders, a main screen image modification module connected to the reconstruction buffer, a second display module, an inappropriate content area calculation module, and a first display module connected to the main screen image modification module; wherein, The first decoder is used to decode the main screen bitstream and generate main screen decoded data; The second decoder is used to decode the secondary screen bitstream and generate secondary screen decoded data; The reconstruction buffer is used to temporarily store the main screen decoding data and the secondary screen decoding data; The first display module is used to send the decoded data of the main screen to the main screen for display; The main screen image modification module is used to modify the area displaying the current display window content of the secondary screen on the main screen to a solid color when the screen isolation function level is warning level, so as to send the main screen decoded data to the main screen for display after information is blocked; The second display module is used to send the decoded data from the secondary screen to the secondary screen for display. The inappropriate content area calculation module is used to calculate the position and size of the secondary screen display window dragged into the main screen when the screen isolation function level is warning level, and submit the calculation result to the main screen image modification module; The level control switch is used by the user to set the forced level, warning level, and disable the screen isolation function. The mouse event limiting module is used to pause the current mouse dragging event when the screen isolation function level is at the warning level and it is determined that the user drags the current display window of the secondary screen onto the main screen.

9. A computer device, comprising: a memory and a processor, wherein the memory stores a computer program, characterized in that... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that... When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images