Interaction control method and device based on screen content, equipment and storage medium

By segmenting the screen of the remote desktop service and coding with different quality, the delay and lag caused by bandwidth limitations in remote desktop service is solved, and the user experience and data transmission efficiency are improved.

CN119065537APending Publication Date: 2024-12-03SHANGHAI JIUCHI NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411049579.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In remote desktop services, especially in application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, direct lossless encoding of the screen can easily exceed the network bandwidth limit, resulting in the display screen on the control side. Delay, lag and disconnection seriously affect the user's user experience.

Method used

By segmenting the screen area, it is divided into an area of ​​interest and a non-interested area, and the two are encoded with different quality based on the actual bandwidth information. The area of ​​interest uses high-quality encoding, and the area of ​​interest uses low Quality encoding ensures smooth data transmission and improves picture quality.

Benefits of technology

While ensuring the smoothness of data transmission, it realizes high-quality display of areas of interest, reduces lag during remote desktop services, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065537A_ABST
    Figure CN119065537A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an interaction control method and device based on screen content, equipment and a storage medium. The interaction control method comprises the following steps: acquiring a currently displayed first screen picture and bandwidth information through first terminal equipment; segmenting the first screen picture to obtain a region-of-interest picture and a non-region-of-interest picture; first coding processing is carried out on the interested area picture, second coding processing is carried out on the non-interested area picture according to the bandwidth information, coded data are obtained, and the picture quality of the first coding processing is higher than that of the second coding processing; sending the coded data to a second terminal device so as to display a second screen picture on the second terminal device according to the coded data; receiving a control instruction generated and sent by the second terminal device in the process of displaying the second screen picture, and updating the first screen picture according to the control instruction; the problem that user experience is poor is solved, high-quality display of images in the region of interest is achieved while data transmission fluency is guaranteed, and user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present application relate to the field of interactive control technology, and in particular, to a method, apparatus, device, and storage medium for interactive control based on screen content. Background Art

[0002] Remote desktop services allow users to access and control remote computers via the Internet anytime and anywhere, thereby achieving remote office and remote technical support, etc. Remote desktop services greatly improve work efficiency, especially for users who need to travel frequently or work from home, remote desktop services are undoubtedly a huge boost.

[0003] In some fields, such as the design field, in application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, these application scenarios have high requirements for the restoration of the screen content of remote desktop services. When performing remote desktop services, if the screen image is directly losslessly encoded and then transmitted, it is easy to exceed the network bandwidth limit, resulting in a series of problems such as delays, freezes and disconnections in the display of the control end, which seriously affects the user experience. Summary of the invention

[0004] The embodiments of the present application provide a screen content-based interactive control method, apparatus, device and storage medium, which can solve the problem of poor user experience during remote desktop service, perform partition coding based on actual bandwidth information, and achieve high-quality display of the screen in the area of ​​interest while ensuring the smoothness of data transmission, thereby improving the user experience when performing remote desktop service.

[0005] In a first aspect, an embodiment of the present application provides a screen content-based interactive control method, which is used in a first terminal device, including:

[0006] Obtain the first screen image currently being displayed and the bandwidth information of the currently connected network;

[0007] Performing regional segmentation on the first screen image to obtain an area of ​​interest image and an area of ​​non-interest image;

[0008] Performing a first encoding process on the picture in the region of interest, and performing a second encoding process on the picture in the region of non-interest according to the bandwidth information, to obtain encoding data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process;

[0009] sending the encoded data to a second terminal device so that a second screen image is displayed on the second terminal device according to the encoded data;

[0010] A control instruction generated and sent by the second terminal device during display of the second screen image is received, and the first screen image is updated according to the control instruction.

[0011] Further, the first screen image is segmented into regions to obtain an area of ​​interest image and an area of ​​non-interest image, including:

[0012] Using a convolutional neural network model, the display content of the first screen image is recognized, and the first screen image is segmented into regions according to the recognition result to obtain an image of an area of ​​interest and an image of an area of ​​non-interest;

[0013] Segmenting a predefined area in the window area corresponding to the target software in the first screen image into an area-of-interest image, and segmenting other areas into non-area-of-interest images;

[0014] The area specified by the area selection instruction sent by the second terminal device is divided into an area of ​​interest screen, and other areas are divided into non-area of ​​interest screens.

[0015] Further, performing a first encoding process on the picture in the region of interest, and performing a second encoding process on the picture in the region of non-interest according to the bandwidth information, to obtain encoded data corresponding to the first screen picture, including:

[0016] Performing a first encoding process on the region-of-interest picture to obtain first encoded data corresponding to the region-of-interest picture;

[0017] Perform content recognition on the non-interest area pictures, perform classification processing based on the recognition results, and obtain content classification results;

[0018] Performing regional segmentation on the non-interest region screen according to the content classification result to obtain sub-region screens corresponding to the content classification result;

[0019] Transmitting the sub-region picture to a corresponding encoder, and controlling the encoder to perform a second encoding process on the sub-region picture according to the bandwidth information to obtain second encoded data;

[0020] The first coded data and the second coded data are integrated to obtain coded data corresponding to the first screen image.

[0021] Further, sending the encoded data to the second terminal device so as to display the second screen image according to the encoded data on the second terminal device includes:

[0022] Comparing the bandwidth information with a first preset bandwidth threshold, and when the bandwidth information exceeds the first preset bandwidth threshold, sending the encoded data to the second terminal device, so that the second terminal device displays a second screen image according to the encoded data;

[0023] When the bandwidth information is below the first preset bandwidth threshold, performing a third encoding process on the first screen image to obtain third encoded data corresponding to the first screen image, wherein the image quality of the third encoding process is lower than the image quality of the second encoding process;

[0024] sending the third coded data to the second terminal device so that the third screen image is displayed on the second terminal device according to the third coded data;

[0025] The first coded data is sent to the second terminal device, so that the second terminal device refreshes and displays the corresponding area picture in the third screen picture according to the first coded data.

[0026] Further, the sub-region screen includes a first type of sub-region screen and a second type of sub-region screen, the first type of sub-region screen includes a text region screen, a line region screen and a pure color picture region screen, and the second type of sub-region screen is the remaining region screens in the non-interest region screen except the first type of sub-region screen;

[0027] The sub-region picture is transmitted to a corresponding encoder, and the encoder is controlled to perform a second encoding process on the sub-region picture according to the bandwidth information to obtain second encoded data, including:

[0028] Transmitting the first type of sub-region pictures to a first encoder, and transmitting the second type of sub-region pictures to a second encoder;

[0029] comparing the bandwidth information with a second preset bandwidth threshold, and when the bandwidth information exceeds the second preset bandwidth threshold, controlling the first encoder to perform fourth encoding processing on the first-type sub-region picture to obtain first sub-encoded data, and the second preset bandwidth threshold is greater than the first preset bandwidth threshold;

[0030] Controlling the second encoder to perform a fifth encoding process on the second type of sub-region picture to obtain second sub-encoded data, wherein the picture quality of the fourth encoding process is higher than the picture quality of the fifth encoding process, the picture quality of the fourth encoding process is equal to or lower than the picture quality of the first encoding process, and the picture quality of the fifth encoding process is higher than the picture quality of the third encoding process;

[0031] The first sub-coded data and the second sub-coded data are integrated to obtain second coded data corresponding to the non-interest region picture.

[0032] Further, after transmitting the first type of sub-region picture to the first encoder and transmitting the second type of sub-region picture to the second encoder, the method includes:

[0033] comparing the bandwidth information with a first preset bandwidth threshold and a second preset bandwidth threshold, and when the bandwidth information is below the second preset bandwidth threshold and exceeds the first preset bandwidth threshold, controlling the first encoder to perform fifth encoding processing on the first-category sub-region picture to obtain third sub-encoded data;

[0034] Controlling the second encoder to perform fifth encoding processing on the second sub-region picture to obtain fourth sub-encoded data;

[0035] The third sub-coded data and the fourth sub-coded data are integrated to obtain second coded data corresponding to the non-interest region picture.

[0036] In a second aspect, an embodiment of the present application provides a screen content-based interactive control method for a second terminal device, comprising:

[0037] Receiving coded data, where the coded data is obtained by performing a first coding process on an area of ​​interest in a first screen image by a first terminal device and performing a second coding process on an area of ​​non-interest in the first screen image according to bandwidth information;

[0038] Decoding the encoded data, rendering the decoded data, and displaying a second screen image, where the second screen image corresponds to the first screen image of the first terminal device;

[0039] receiving an interactive signal during the display of the second screen image, and generating a control instruction in response to the interactive signal;

[0040] The control instruction is sent to the first terminal device, so that the first terminal device updates the first screen image according to the control instruction.

[0041] In a third aspect, an embodiment of the present application provides an interactive control device based on screen content, which is used in a first terminal device, including:

[0042] An information acquisition module, used to acquire the first screen image currently displayed and the bandwidth information of the currently connected network;

[0043] A picture segmentation module, used for performing regional picture segmentation on the first screen picture to obtain pictures of interest areas and pictures of non-interest areas;

[0044] The encoding module is used to perform a first encoding process on the picture in the region of interest, and perform a second encoding process on the picture in the region of non-interest according to the bandwidth information, so as to obtain encoding data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process;

[0045] A data sending module, used for sending the coded data to a second terminal device, so that the second terminal device displays a second screen image according to the coded data;

[0046] The instruction receiving module is used to receive the control instruction generated and sent by the second terminal device during the process of displaying the second screen image, and update the first screen image according to the control instruction.

[0047] In a fourth aspect, an embodiment of the present application provides an interactive control device based on screen content, which is used in a second terminal device, including:

[0048] A data receiving module, configured to receive coded data, wherein the coded data is obtained by performing a first coding process on an area of ​​interest in a first screen image by a first terminal device and performing a second coding process on an area of ​​non-interest in the first screen image according to bandwidth information;

[0049] A decoding and rendering module, used to decode the encoded data, render the decoded data, and display a second screen image, where the second screen image corresponds to the first screen image of the first terminal device;

[0050] A control instruction generating module, used for receiving an interactive signal during the process of displaying the second screen image, and generating a control instruction in response to the interactive signal;

[0051] The instruction sending module is used to send the control instruction to the first terminal device, so that the first terminal device updates the first screen image according to the control instruction.

[0052] In a fifth aspect, an embodiment of the present application provides an interactive control device based on screen content, including:

[0053] memory and one or more processors;

[0054] A memory for storing one or more programs;

[0055] When one or more programs are executed by one or more processors, the one or more processors implement the screen content-based interactive control method of the first aspect or the second aspect.

[0056] In a sixth aspect, an embodiment of the present application provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the screen content-based interactive control method of the first aspect or the second aspect.

[0057] The embodiment of the present application is used for a first terminal device, by acquiring a first screen picture currently displayed and bandwidth information of a current connection, performing regional screen segmentation on the first screen picture to obtain an interested region picture and a non-interested region picture, performing a first encoding process on the interested region picture, performing a second encoding process on the non-interested region picture according to the bandwidth information, obtaining encoded data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process, sending the encoded data to a second terminal device, so that the second terminal device displays a second screen picture according to the encoded data, receiving a control instruction generated and sent by the second terminal device in the process of displaying the second screen picture, and updating the first screen picture according to the control instruction. By adopting the above technical means, the first screen image can be divided into the image of the region of interest and the image of the region of non-interest, the image of the region of interest is encoded based on a higher image quality, and the image of the region of non-interest is encoded according to the bandwidth information to adapt to the bandwidth information of the current connection network, so that the image of the region of interest can be displayed in high quality on the second screen of the second terminal device while ensuring the smoothness of data transmission, thereby avoiding the problem of poor user experience during remote desktop service, and based on the high-quality display of the image of the region of interest on the second screen, the user's viewing experience is improved, thereby improving the overall user experience during the use of remote desktop service. In addition, the image of the region of non-interest is encoded according to the bandwidth information to adapt to the bandwidth information of the current connection network, which can ensure the smoothness of data transmission, thereby reducing the jamming during remote desktop service, and thus improving the overall user experience during the use of remote desktop service. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of terminal device connection for a remote desktop service provided in an embodiment of the present application;

[0059] Figure 2 is a flow chart of an interactive control method based on screen content provided by an embodiment of the present application;

[0060] Figure 3 This is a schematic diagram of dividing the region of interest of a target software provided in an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of dividing the region of interest of another target software provided in an embodiment of the present application;

[0062] Figure 5 is a flowchart of another interactive control method based on screen content provided in an embodiment of the present application;

[0063] Figure 6It is a schematic diagram of a frame selection based on a second screen image provided by an embodiment of the present application;

[0064] Figure 7 is another schematic diagram of frame selection based on the second screen provided in an embodiment of the present application;

[0065] Figure 8 is a schematic diagram of a region of interest after the screen is reduced, provided in an embodiment of the present application;

[0066] Fig. 9 is a structural schematic diagram of an interactive control device based on screen content provided in an embodiment of the present application;

[0067] Fig.10 is a structural schematic diagram of an interactive control device based on screen content provided in an embodiment of the present application;

[0068] Fig.11 It is a structural diagram of a screen content-based interactive control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0070] In some fields, such as the design field, in application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, the degree of restoration of the screen content of the remote desktop service is relatively high based on these application scenarios. When performing remote desktop services, if the screen image is directly losslessly encoded and then transmitted, it is easy to exceed the network bandwidth limit, resulting in a series of problems such as delay, freeze and disconnection of the control end display image, which seriously affects the user experience. If the encoding process is directly performed according to the actual bandwidth information, when the bandwidth is small, the screen image is encoded with low picture quality to adapt to the bandwidth of the actual connection network, the overall picture quality of the screen image (i.e., the remote desktop image) displayed by the receiving device will be low. For application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, the degree of restoration of the screen content of the remote desktop service is relatively high based on these application scenarios. The low picture quality of the remote desktop image display will seriously affect the user's viewing, thereby affecting the corresponding work process.

[0071] In order to improve the user experience, classified encoding is performed for content with different characteristics. Different types of content use different encoders to encode different picture qualities. For example, text, lines, and solid color image content use lossless encoding (i.e., high-quality encoding), while other images or videos use lossy encoding (i.e., low-quality encoding), such as JPEG image encoding and H26.4 video encoding. However, content classification often only focuses on content, while ignoring the content area that users actually care about. For example, when a user zooms in and fine-tunes a rendered image of a designed device, even if the rendered image is classified as an image and lossy encoding (i.e., low-quality encoding) is performed based on the classification result, the user expects the content in this area to be displayed with high quality so that the user can judge whether the modification result of the design drawing meets expectations. Therefore, the existing scheme for encoding different picture qualities based on the content classification results also has the problem of poor user experience.

[0072] Based on this, an interactive control method based on screen content of an embodiment of the present application is provided, which aims to divide the first screen into an area of ​​interest and an area of ​​non-interest during the remote desktop service process, encode the area of ​​interest based on a higher picture quality, and encode the area of ​​non-interest according to the bandwidth information to adapt to the bandwidth information of the current connection network, so that the area of ​​interest can be displayed in high quality on the second screen of the second terminal device while ensuring the smoothness of data transmission, thereby avoiding the problem of poor user experience during remote desktop service. Compared with the existing full-screen lossless encoding or full-screen lossy encoding or content classification encoding, this embodiment can ensure the smoothness of data transmission while achieving high-quality display of the area of ​​interest on the second screen of the second terminal device, thereby improving the user's viewing experience, thereby improving the user's overall experience during the use of remote desktop services. In addition, the area of ​​non-interest is encoded according to the bandwidth information to adapt to the bandwidth information of the current connection network, which can ensure the smoothness of data transmission, thereby reducing the jamming during remote desktop services, and thus improving the user's overall experience during the use of remote desktop services.

[0073] Figure 1 This is a schematic diagram of a terminal device connection for a remote desktop service provided in an embodiment of the present application. Figure 1 In the remote desktop service process, the first terminal device 10, as the controlled terminal device, can be a high-performance terminal device, such as a computer or cloud server with a high-performance design. The second terminal device 20, as the user end, can also be called the control end, such as a mobile terminal, a smart tablet, and a computer device with low performance configuration. The user can establish a communication connection with the first terminal device 10 and perform the corresponding remote desktop service based on the remote desktop service-related software in the second terminal device 20. The first terminal device 10 collects and encodes its own first screen image to obtain encoded data, and transmits the encoded data to the second terminal device 20. After receiving the encoded data, the second terminal device 20 decodes and renders the encoded data to display the second screen image. It should be noted that the second screen image and the first screen image are the same in the user's vision, but in fact there is a difference in image quality between the two. The second terminal device 20 generates a corresponding control instruction based on displaying the second screen image, and transmits the control instruction to the first terminal device 10. The first terminal device 10 receives the control instruction and updates the first screen image according to the control instruction, thereby achieving the effect of controlling the first screen image corresponding to the first terminal device 10 on the second terminal device 20, that is, achieving the effect of remote desktop control.

[0074] In one embodiment, when the first terminal device is a cloud server, the first screen image is a desktop image of the cloud server, also known as a cloud desktop, desktop virtualization or cloud computer, which is an application form of cloud computing technology.

[0075] Figure 2 A flow chart of an interactive control method based on screen content provided in an embodiment of the present application is given. The interactive control method based on screen content provided in this embodiment can be executed by an interactive control device based on screen content. The interactive control device based on screen content can be implemented by software and / or hardware. The interactive control device based on screen content can be composed of two or more physical entities or one physical entity. Generally speaking, the interactive control device based on screen content can be a terminal device, such as a computer device or a server.

[0076] The following description is made by taking a computer device or a server as an example to implement the interactive control method based on screen content. Figure 2 The interactive control method based on screen content is used for a first terminal device, and the interactive control method based on screen content specifically includes:

[0077] S101: Obtain the bandwidth information of the currently displayed first screen image and the currently connected network.

[0078] In some fields, such as design, such as 3D modeling and rendering, architectural design and CAD, animation and video editing and other application scenarios, these application scenarios usually involve complex modeling, rendering and animation production processes, and these design tasks require a lot of computing resources. Therefore, computer devices with ordinary functional configurations may not be able to meet this high-performance demand, so high-performance configuration computer devices or cloud computers are needed to meet this high-performance demand. Cloud computers can be understood as cloud servers, which are virtual computer services based on cloud computing technology. It allows users to obtain and use complete computer operating systems, applications, storage and data resources on remote cloud servers through network connections, as if they were using a real computer. Cloud computers encapsulate traditional computer hardware resources (such as CPU, memory, hard disk, etc.) on cloud servers, and allocate these resources to different users through virtualization technology, so that each user can obtain an independent virtual computer environment.

[0079] Generally, the user end is configured as a computer device with ordinary functions. In order to improve the user's design work efficiency, remote desktop control can be performed on the corresponding computer device with higher performance configuration or cloud computer (i.e., the first terminal device in this embodiment) through the computer device with ordinary configuration on the user end (i.e., the second terminal device in this embodiment), so as to realize remote control of the computer device with higher performance configuration or cloud computer (i.e., the first terminal device in this embodiment) to perform complex design work such as 3D modeling and rendering, architectural design and CAD, animation and video editing.

[0080] After establishing a communication connection between the first terminal device and the second terminal device, the first terminal device obtains the bandwidth information of the currently displayed first screen image and the currently connected network, and then performs corresponding regional screen segmentation and encoding processing based on the first screen image to generate encoded data corresponding to the first screen image, and transmits it to the second terminal device. As described above, by obtaining bandwidth information, corresponding processing can be performed based on the bandwidth information to obtain encoded data that meets the display quality requirements and adapts to the bandwidth information, so as to ensure the smooth progress of the subsequent remote desktop service process, avoid delays and freezes, and improve the picture quality as much as possible based on the bandwidth information, thereby improving the user experience.

[0081] In one embodiment, when the first terminal device is a cloud server, the first screen image is a desktop image of the cloud server, also known as a cloud desktop, desktop virtualization or cloud computer.

[0082] S102: Segment the first screen image into regions to obtain an image of an area of ​​interest and an image of an area of ​​non-interest.

[0083] For application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, the user focuses on a certain area for viewing or designing, and this area can be understood as the user's area of ​​interest. Therefore, after the first terminal device obtains the first screen image, the first screen image is segmented to obtain the area of ​​interest image and the area of ​​non-interest image. Subsequently, the area of ​​interest image and the area of ​​non-interest image can be encoded separately to achieve high-quality display of the area of ​​interest in the second terminal device, so that the area of ​​interest that the user focuses on can be displayed in high quality, thereby improving the user's viewing experience and further improving the user's overall design efficiency.

[0084] When the regional screen is segmented, the display content of the first screen can be identified by a convolutional neural network model, such as a CNN recognition model, and the first screen is segmented according to the recognition result to obtain the region of interest and the non-region of interest. Exemplarily, the local feature extraction of the first screen is performed by the convolution layer in the convolutional neural network model to generate a feature map, which contains key information such as edges, textures and shapes in the image (first screen). In the target detection task, the convolutional neural network model will identify the region of interest (ROI) that may contain the target on the feature map. This is achieved by sliding a window on the feature map or using a regional proposal network. Each window or proposal area will be further processed to determine whether it contains the target and the category and location of the target. For each region of interest, the convolutional neural network model will crop or pool the corresponding feature map part to extract the feature representation of the region. Then, these feature representations will be input into the fully connected layer for classification processing. The fully connected layer will calculate the probability that the region belongs to each category based on the learned weights and biases, and output the final classification result. In the target detection task, in addition to classification, the precise position of the target also needs to be determined. The precise position of the target can be achieved through bounding box regression, that is, the convolutional neural network model will learn a mapping relationship, which is a mapping relationship from feature representation to bounding box coordinates. According to the learned mapping relationship, the corresponding feature representation (corresponding to the target's region of interest) can be determined as the target's bounding box coordinates, so that the target's bounding box coordinates can be output while outputting the classification results. According to the target's bounding box coordinates, the final segmentation results of the region of interest screen and the non-region of interest screen can be obtained. As described above, the convolutional neural network model can automatically identify and segment the region of interest screen and the non-region of interest screen, thereby improving the work efficiency of regional screen segmentation, thereby improving the overall work efficiency of interactive control, and improving the user experience.

[0085] When the regional screen is divided, the predefined area in the window area corresponding to the target software in the first screen can be divided into the area of ​​interest screen, and the other areas can be divided into the area of ​​non-interest screen. The target software can be understood as design software, such as 3D modeling software, rendering software, architectural design software, CAD software, and animation and video editing software. The corresponding area of ​​interest range in the corresponding target software is predefined, that is, the predefined area in the window area corresponding to each target software is predefined, and the predefined area is the preset area of ​​interest. When the user opens any corresponding target software later, the area of ​​interest screen and the area of ​​non-interest screen in the window area of ​​the corresponding target software can be directly determined according to the corresponding predefined area. Figure 3This is a schematic diagram of the region of interest division of a target software provided in an embodiment of the present application, referring to Figure 3 , assuming that the current first screen image is the window area A0 corresponding to software A, and assuming that area A10 within the window area A0 is predefined as a predefined area (i.e., an area of ​​interest), then after opening software A, area A10 in the software A is determined to be an area of ​​interest based on the predefined area, and area A10 in the window area of ​​software A is divided into an area of ​​interest image, and the remaining area A20 is divided into a non-area of ​​interest image. Figure 4 This is another schematic diagram of the region of interest division of target software provided in the embodiment of the present application, referring to Figure 4 , assuming that the current first screen image is the window area B0 corresponding to software B, assuming that the area B10 in the window area B0 is predefined as a predefined area (i.e., an area of ​​interest), after opening software B, the area B10 in the software B is determined as an area of ​​interest based on the predefined area, and the area B10 in the window area B0 of the software B is segmented into an area of ​​interest screen, and the remaining area B20 is segmented into an area of ​​non-interest screen. As described above, by automatically identifying and segmenting the area of ​​interest screen and the area of ​​non-interest screen in the window area of ​​the target software, the work efficiency of area screen segmentation is improved, thereby improving the overall work efficiency of interactive control, and improving the user experience.

[0086] When the regional screen is divided, the first screen image can be divided into regional screens according to the areas selected by the user. The second terminal device generates a corresponding area selection instruction based on the user's selection of the displayed second screen image (that is, the first screen image corresponds to the screen image displayed on the second terminal device), and transmits it to the first terminal device. The first terminal device receives the area selection instruction sent by the second terminal device, and divides the area specified by the area selection instruction (the area in the first screen image) into an area of ​​interest screen, and divides the remaining areas into non-area of ​​interest screens. As described above, by dividing the first screen image into an area of ​​interest screen and a non-area of ​​interest screen through user selection, the corresponding area of ​​interest screen can be divided based on the user's wishes, thereby improving the user experience.

[0087] It should be noted that, in the actual implementation process, the first terminal device can perform regional screen segmentation processing based on one of the three segmentation methods mentioned above.

[0088] S103, performing a first encoding process on the picture in the region of interest, and performing a second encoding process on the picture in the region of non-interest according to the bandwidth information, to obtain encoding data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process.

[0089] In the design field, for example, in application scenarios such as 3D modeling and rendering, architectural design and CAD, animation and video editing, these application scenarios have high requirements for the restoration of the screen content of the remote desktop service, but usually the restoration of the screen content of the area that needs to be improved is high. Even if the image quality of the remaining areas is poor, it does not affect the current design improvement work. The screen of the area of ​​current design improvement can be understood as the aforementioned area of ​​interest screen. Therefore, after the first screen image is divided into the area of ​​interest screen and the area of ​​non-interest screen, different picture quality encoding processing can be performed on the area of ​​interest screen and the area of ​​non-interest screen, respectively, to ensure that the area of ​​interest screen corresponds to the first encoding processing with high picture quality, so that after subsequent transmission to the second terminal device, in the screen image displayed by the second terminal device (i.e., the second screen image), the corresponding area of ​​interest realizes high picture quality display. Compared with the existing method of high picture quality encoding or low picture quality encoding of the entire picture, this embodiment can realize high picture quality display of the area of ​​interest, thereby improving the user's viewing experience, and then improving the overall design work efficiency. Exemplarily, when the first terminal device performs encoding, the first encoding process can be performed on the image of the region of interest to obtain the first encoded data corresponding to the image of the region of interest, and the second encoding process can be performed on the image of the non-interest region according to the bandwidth information to obtain the second encoded data corresponding to the image of the non-interest region. Among them, the image quality of the first encoding process is higher than the image quality of the second encoding process, so that after the encoded data is subsequently transmitted to the second terminal device, when the corresponding second screen image is displayed on the second terminal device, the image of the region of interest corresponds to a high image quality display, thereby improving the user's viewing experience, and further improving the overall work efficiency of the design work.

[0090] In the non-interest area screen, there may be a variety of content elements, such as text, lines, icons, solid color pictures, images and videos, etc. Based on the different requirements of different content elements for picture quality, for example, text, lines and solid color pictures have lower requirements for picture quality, and can display the corresponding text, lines and picture colors. Therefore, the non-interest area screen can be identified, and classified according to the identification result to obtain the content classification result. The non-interest area screen is segmented according to the content classification result to obtain the sub-area screen of the corresponding content classification result. Exemplarily, the sub-area screen includes a first type of sub-area screen and a second type of sub-area screen, the first type of sub-area screen includes a text area screen, a line area screen and a solid color picture area screen, and the second type of sub-area screen is the remaining area screen in the non-interest area screen except the first type of sub-area screen. The sub-area screen is transmitted to the corresponding encoder, and the encoder is controlled to perform a second encoding process on the sub-area screen according to the bandwidth information to obtain the second encoded data adapted to the bandwidth information. The first encoded data corresponding to the interest area screen and the second encoded data corresponding to the non-interest area screen are integrated to obtain the encoded data corresponding to the first screen screen.

[0091] In one embodiment, the first encoding process may be a lossless encoding process so that the image in the region of interest is displayed as high quality as possible; the second encoding process may be a lossy encoding process so that the image in the region of non-interest is displayed at relatively low quality.

[0092] In the above, the first screen image is divided and segmented into an area of ​​interest image and an area of ​​non-interest image, and a first encoding process with high picture quality is performed on the area of ​​interest image to obtain first encoded data, and a second encoding process with low picture quality is performed on the area of ​​non-interest image to obtain second encoded data, and the first encoded data and the second encoded data are integrated to obtain encoded data corresponding to the first screen image. After the encoded data is subsequently transmitted to the second terminal device, the corresponding screen image (i.e., the second screen image) is displayed on the second terminal device, and the corresponding area of ​​interest image in the screen image (i.e., the second screen image) can be displayed with high picture quality, so that the area of ​​user concern can be displayed with high picture quality, which is helpful for user viewing, thereby improving the user's viewing experience, and further improving the overall work efficiency of user viewing and design.

[0093] S104: Send the encoded data to the second terminal device, so that the second terminal device displays a second screen image according to the encoded data.

[0094] The coded data corresponding to the first screen is sent to the second terminal device, so that the second screen is displayed according to the coded data on the second terminal device. The second screen is visually identical to the first screen, and the substantial difference is that the picture quality is different. The coded data is obtained by encoding the first screen in the first terminal device, and sent to the second terminal device, so that the second terminal device can display the second screen corresponding to the first screen, that is, realize the function of remote desktop display, and the user can perform corresponding viewing and interactive control based on the second screen displayed by the second terminal device. In the use scenario with high requirements for picture quality, the design screen (i.e., the first screen) with high requirements for corresponding performance configuration can be viewed on the second terminal device with low performance configuration, and control interaction can be performed, thereby improving the user experience.

[0095] In one embodiment, the bandwidth information based on the current connection network is an important factor affecting the transmission of coded data. When the bandwidth is small, if the transmitted coded data is large, it will cause the second terminal device to receive the coded data with a delay or freeze, which will cause the second terminal device to display the second screen image with a corresponding delay or freeze, thereby affecting the overall user experience. Therefore, before sending the coded data, the first terminal device first compares the acquired bandwidth information of the current connection network with the first preset bandwidth threshold, and performs corresponding coding and transmission control according to the comparison result. When the bandwidth information exceeds the first preset bandwidth threshold, it means that the bandwidth is large at this time, and the amount of coded data that can be transmitted is large, so the coded data corresponding to the first screen image obtained by the above integration can be directly sent to the second terminal device. When the bandwidth information is below the first preset bandwidth threshold, it means that the bandwidth is small at this time. If the coded data corresponding to the first screen image obtained by the above transmission is directly sent to the second terminal device, it is possible that due to the small bandwidth, the second terminal device will receive the coded data with a delay and display the coded data. There will be freezes. Therefore, the first screen image can be subjected to a third coding process as a whole to obtain the third coded data corresponding to the first screen image, wherein the picture quality of the third coding process is lower than the picture quality of the second coding process. The picture quality corresponding to the third coding process can be understood as a lower picture quality. The first terminal device first sends the third coded data to the second terminal device, so that the second terminal device displays the third screen picture according to the third coded data. The third screen picture can be understood as a first screen picture with low picture quality, that is, the third screen picture has a lower picture quality than the second screen picture. The second terminal device displays the third screen picture for the user to view initially, so as to avoid the problem that the screen picture cannot be loaded for a long time due to the small bandwidth, which makes the user experience poor. On the basis of the second terminal device displaying the third screen information, the first terminal device sends the first coded data corresponding to the interested area picture to the second terminal device, so that the second terminal device refreshes and displays the corresponding area picture in the third screen picture according to the first coded data, so as to realize the gradual refreshing of the interested area picture to high picture quality in the third screen picture with low picture quality, so that the subsequent user can watch and interact based on the interested area with high picture quality when designing based on the concerned area (i.e., the interested area picture), thereby improving the user experience. In the process of gradually refreshing the interested area picture to high picture quality in the third screen picture with low picture quality, it is imperceptible to the user's perception, which further improves the user experience.

[0096] In the above, different coded data transmission controls are respectively performed according to the size of the bandwidth information. When the bandwidth is large, the first coding processing with high picture quality is performed on the picture of the region of interest, and the second coding processing with lower picture quality is performed on the picture of the non-interest region. The coding amount of the picture of the non-interest region is reduced, so that the overall coding data amount is reduced, thereby improving the smoothness of the coded data transmission. In addition, the first coding processing with high picture quality is performed based on the picture of the region of interest. It can be guaranteed that the picture of the region of interest with high picture quality is subsequently displayed in the second terminal device based on the coded data, which can ensure the high picture quality display of the user's attention area, thereby improving the user experience. When the bandwidth is small, the first screen image is first encoded with the lowest picture quality as a whole, so that the second terminal device can display the third screen image first, avoiding the screen loading jam caused by the small bandwidth and affecting the user experience. Although the picture quality of the third screen image is low, from the user's point of view, this embodiment can also quickly display the remote desktop (i.e., the third screen image) when the bandwidth is small, thereby improving the user experience; after displaying the third screen image with lower picture quality, the first encoded data corresponding to the image of the area of ​​interest is sent to the second terminal device, so that the second terminal device can gradually refresh the corresponding image of the area of ​​interest to high picture quality based on the first encoded data, so that subsequent users can view the image with high picture quality (i.e., the image of the area of ​​interest) when viewing and interactively controlling the image of the area of ​​interest, thereby further improving the user experience.

[0097] S105: Receive a control instruction generated and sent by the second terminal device during the process of displaying the second screen image, and update the first screen image according to the control instruction.

[0098] By encoding the first screen image in the first terminal device to obtain the encoded data, and sending the encoded data to the second terminal device, the second terminal device can display the second screen image corresponding to the first screen image, that is, realize the function of remote desktop display, the user can view and interact accordingly based on the second screen image displayed by the second terminal device, generate corresponding control instructions, and feedback the control instructions to the first terminal device, the first terminal device updates the first screen image based on the control instructions, and re-executes S101-S104 on the updated first screen image, so that the second terminal device can receive the encoded data corresponding to the updated first screen image and display the updated second screen image. Through the interactive control of the first terminal device and the second terminal device mentioned above, the effect of controlling the first screen image corresponding to the first terminal device on the second terminal device is realized, that is, the effect of remote desktop control is realized, so that high-performance work, such as complex design work, can be performed on the second terminal device with relatively low performance configuration, thereby saving overall resource configuration, reducing the cost investment of the user end, and thus improving the overall user experience.

[0099] In one embodiment, the second terminal device generates and sends a control instruction during the process of displaying the second screen image, wherein the control instruction includes a user selecting a frame on the displayed second screen image (i.e., the first screen image corresponds to the screen image displayed on the second terminal device) to generate a corresponding area selection instruction.

[0100] In one embodiment, when performing the second encoding process, the encoding rate of the corresponding encoder can be adjusted according to the bandwidth information, so that the second encoded data obtained by encoding can adapt to the current bandwidth information, avoiding delays and freezes caused by bandwidth reasons in the subsequent transmission process. Exemplarily, the sub-region screen includes a first-type sub-region screen and a second-type sub-region screen, the first-type sub-region screen includes a text region screen, a line region screen and a solid color picture region screen, and the second-type sub-region screen is the remaining region screen in the non-interest region screen except the first-type sub-region screen. The first-type sub-region screen can be transmitted to the first encoder, and the second-type sub-region screen can be transmitted to the second encoder. In order to further improve the adaptability of the encoded data to the bandwidth, different encoding processes can be performed on the non-interest region screen according to the size of the bandwidth. After obtaining the bandwidth information of the current connection network, the bandwidth information is compared with the second preset bandwidth threshold. Among them, the second preset bandwidth threshold is greater than the first preset bandwidth threshold. When the bandwidth information exceeds the second preset bandwidth threshold, that is, when the bandwidth information exceeds the first preset bandwidth threshold and also exceeds the second preset bandwidth threshold, it means that the bandwidth is large and the amount of encoded data that can be transmitted is also large. Therefore, on the basis of the first encoding process of the interest area picture with high picture quality to obtain the corresponding first encoded data; for the non-interest area picture, the first encoder is controlled to perform the fourth encoding process on the first type of sub-area picture to obtain the first sub-encoded data. Among them, the picture quality of the fourth encoding process is equal to or lower than the picture quality of the first encoding process. By performing high-quality encoding process on the first type of sub-area picture, the picture quality of the overall picture is improved as much as possible on the basis of adapting to the bandwidth information to achieve smooth transmission, and the user experience is further improved. For the second type of sub-area picture, the second encoder can be controlled to perform the fifth encoding process on the second type of sub-area picture to obtain the second sub-encoded data. Among them, the picture quality of the fourth encoding process is higher than the picture quality of the fifth encoding process. It can be understood that the picture quality of the fifth encoding process is lower than the picture quality of the fourth encoding process. Therefore, even if the picture quality of the fourth encoding process is the same as the picture quality of the first encoding process, the picture quality of the second encoded data of the non-interest area picture obtained by integrating the first sub-encoded data and the second sub-encoded data is lower than the picture quality of the first encoded data. At the same time, the picture quality of the fifth encoding processing is higher than that of the third encoding processing. Based on the bandwidth information exceeding the first preset threshold (and exceeding the second preset threshold), the second type of sub-area picture does not require as low picture quality as the third encoding processing (the encoding processing when the bandwidth information is below the first preset threshold). Therefore, the picture quality of the fifth encoding processing can be set higher than the picture quality of the third encoding processing, so that the picture quality of the corresponding second encoded data of the non-interest area picture can be improved as much as possible on the basis of adapting to the bandwidth information.Integrate the first coded data corresponding to the area of ​​interest picture and the second coded data corresponding to the area of ​​non-interest picture to obtain coded data corresponding to the first screen picture, and send the coded data to the second terminal device so that the second terminal device displays the second screen picture according to the coded data.

[0101] In the above, when the bandwidth information is large, that is, when the bandwidth information exceeds the second preset threshold, the fourth encoding processing with higher picture quality is performed on the first type of sub-region pictures in the non-interest area picture, and the fifth encoding processing with relatively lower picture quality is performed on the second type of sub-region pictures, so as to improve the picture quality of the non-interest area picture as much as possible on the basis of adapting to the bandwidth information, so that the final encoded data corresponding to the first screen picture is transmitted to the second terminal device, and the picture quality of the second screen picture displayed by the second terminal device according to the encoded data is as high as possible, thereby improving the user's viewing experience, and at this time the bandwidth is large, which greatly reduces the probability of delay or freeze, further improving the user's viewing experience.

[0102] In one embodiment, when the bandwidth information is below the second preset bandwidth threshold and exceeds the first preset bandwidth threshold, the bandwidth at this time is also large, but smaller than the second preset bandwidth threshold. Therefore, on the basis of ensuring the first encoding processing based on the higher picture quality of the interested region picture, for the non-interest region picture, the first encoder is controlled to perform the fifth encoding processing on the first type of sub-region picture to obtain the third sub-encoded data, and the second encoder is controlled to perform the fifth encoding processing on the second type of sub-region picture to obtain the fourth sub-encoded data. At this time, the third sub-encoded data and the fourth sub-encoded data can be understood as encoded data with the same picture quality. The picture quality of the fifth encoding processing is higher than the picture quality of the third encoding processing. Based on the bandwidth information exceeding the first preset threshold (but below the second preset threshold), the second type of sub-region picture does not need as low a picture quality as the third encoding processing (encoding processing when the bandwidth information is below the first preset threshold). Therefore, the fifth encoding processing can be set to have a higher picture quality than the third encoding processing, so that the picture quality of the non-interest region picture can be guaranteed as much as possible on the basis of adapting to the bandwidth information. The third sub-encoded data and the fourth sub-encoded data are integrated to obtain the second encoded data corresponding to the non-interest region picture. It should be noted that the picture quality corresponding to the second coded data at this time is lower than the picture quality of the second coded data when the bandwidth information exceeds the second preset threshold, so as to avoid the delay or jamming caused by insufficient bandwidth to transmit the second coded data with higher picture quality. In the above, by performing the fifth coding process on the first sub-region picture and the second sub-region picture corresponding to the non-interest area picture when the bandwidth information is below the second preset bandwidth threshold and exceeds the first preset bandwidth threshold, the second coded data corresponding to the non-interest area picture is integrated, and compared with when the bandwidth information exceeds the second preset bandwidth threshold, the picture quality of the second coded data corresponding to the non-interest area picture is relatively low (but higher than the picture quality of the third coded data), and the bandwidth information can be adapted to transmit the coded data, which can reduce the probability of delay or jamming, thereby improving the smoothness of data transmission of the remote desktop service, avoiding the delay or jamming of the second terminal device, and thus improving the user experience. In addition, based on the first coded data corresponding to the area of ​​interest picture having the highest picture quality, it can be guaranteed that the area of ​​interest picture can be displayed with high picture quality on the second terminal device, further improving the user experience. Integrate the first coded data corresponding to the area of ​​interest picture and the second coded data corresponding to the area of ​​non-interest picture to obtain coded data corresponding to the first screen picture, and send the coded data to the second terminal device so that the second terminal device displays the second screen picture according to the coded data.

[0103] In one embodiment, the first encoder and the second encoder are both composed of a plurality of sub-encoders, and the corresponding sub-region images are transmitted to the corresponding sub-encoders for corresponding encoding processing according to the content classification results. For example, the first encoder includes a first sub-encoder, a second sub-encoder, and a third sub-encoder; the text region image is transmitted to the first sub-encoder, the line region image is transmitted to the second sub-encoder, and the solid color image region image is transmitted to the third sub-encoder. Each sub-encoder performs a second encoding processing on the received sub-region image according to the bandwidth information to obtain second encoded data corresponding to the non-interest region image.

[0104] The above is used for a first terminal device, by acquiring the currently displayed first screen image and the bandwidth information of the current connection, performing regional screen segmentation on the first screen image to obtain an interested area image and a non-interested area image, performing a first encoding process on the interested area image, performing a second encoding process on the non-interested area image according to the bandwidth information, obtaining encoded data corresponding to the first screen image, wherein the image quality of the first encoding process is higher than the image quality of the second encoding process, sending the encoded data to a second terminal device, so that the second terminal device displays the second screen image according to the encoded data, receiving a control instruction generated and sent by the second terminal device during the display of the second screen image, and updating the first screen image according to the control instruction. By adopting the above technical means, the first screen image can be divided into the image of the region of interest and the image of the region of non-interest, the image of the region of interest is encoded based on a higher image quality, and the image of the region of non-interest is encoded according to the bandwidth information to adapt to the bandwidth information of the current connection network, so that the image of the region of interest can be displayed in high quality on the second screen of the second terminal device while ensuring the smoothness of data transmission, thereby avoiding the problem of poor user experience during remote desktop service, and based on the high-quality display of the image of the region of interest on the second screen, the user's viewing experience is improved, thereby improving the overall user experience during the use of remote desktop service. In addition, the image of the region of non-interest is encoded according to the bandwidth information to adapt to the bandwidth information of the current connection network, which can ensure the smoothness of data transmission, thereby reducing the jamming during remote desktop service, and thus improving the overall user experience during the use of remote desktop service.

[0105] On the other hand, refer to Figure 5 , another interactive control method based on screen content of an embodiment of the present application is provided. The interactive control method based on screen content is used for a second terminal device, and the second terminal device may be an electronic device such as a mobile terminal device (such as a smart phone), a tablet computer, and a computer device. The interactive control method based on screen content specifically includes:

[0106] S201. Receive coded data, where the coded data is obtained by performing a first coding process on an area of ​​interest in a first screen image by a first terminal device and performing a second coding process on an area of ​​non-interest in the first screen image according to bandwidth information.

[0107] After establishing a communication connection between the first terminal device and the second terminal device, the first terminal device generates coded data corresponding to the first screen image through the aforementioned S101-S104 coding, and sends the coded data to the second terminal device. The second terminal device continues to receive the coded data transmitted by the first terminal device according to the bandwidth information of the currently connected network, and subsequently displays the corresponding second screen image based on the coded data. The second screen image has the same display content as the first screen image, and the difference is that there is a difference in image quality.

[0108] S202: Decode the encoded data, render the decoded data, and display a second screen image, where the second screen image corresponds to the first screen image of the first terminal device.

[0109] The second terminal device decodes the encoded data, renders the decoded data, and displays a second screen image, which corresponds to the first screen image of the first terminal device. The second terminal device receives the encoded data sent by the first terminal device, decodes the encoded data, and renders and displays the second screen image corresponding to the first screen image, thereby realizing the remote desktop display function, so that the desktop image (also called cloud desktop) on the first terminal device can be displayed on the second terminal device, avoiding the need for high-performance configuration on the second terminal device to display the corresponding first screen image, thereby reducing the cost investment of the second terminal device, and improving the user experience based on the ability to remotely view the first screen image.

[0110] S203: Receive an interactive signal during the process of displaying the second screen image, and generate a control instruction in response to the interactive signal.

[0111] When the second terminal device is displaying the second screen, the user may input a corresponding interaction signal based on the interaction with the second terminal device, such as an interaction signal generated by performing an interaction operation such as clicking, zooming or switching the second screen. The second terminal device receives the interaction signal during the display of the second screen, and generates a control instruction in response to the interaction signal. By generating a control instruction based on the interaction signal during the display of the second screen by the second terminal device, the first screen in the first terminal device may be updated based on the control instruction, thereby realizing the function of the remote desktop control service, so that the first screen with a higher performance configuration requirement can also be updated on the second terminal device with a lower performance configuration, thereby improving the user experience.

[0112] In one embodiment, the control instructions generated during the display of the second screen include instructions such as click, zoom, delete, add, switch and area selection instructions. The area selection instruction is a frame selection operation on the area of ​​interest screen. The second terminal device generates a corresponding area selection instruction based on the user's frame selection of the displayed second screen (that is, the first screen corresponds to the screen displayed on the second terminal device), and transmits it to the first terminal device. The first terminal device receives the area selection instruction sent by the second terminal device, and divides the area specified by the area selection instruction (the area in the first screen) into an area of ​​interest screen, and divides the remaining area into an area of ​​non-interest screen. The area selection instruction generated by the user's frame selection divides the second screen into an area of ​​interest screen and an area of ​​non-interest screen, so that the first terminal device can divide the first screen into an area of ​​interest screen and an area of ​​non-interest screen, and realizes the segmentation of the corresponding area of ​​interest screen based on the user's wishes, thereby improving the user's experience. Figure 6 is a schematic diagram of a frame selection based on a second screen image provided by an embodiment of the present application, referring to Figure 6 , assuming that the second screen image (ie, the first screen image) is a design page of a corresponding design software, the design page includes multiple element areas, such as Figure 6Element area C10, element area C11, element area C12 and element area C13 in the image. When in the frame selection mode (which can be triggered by the frame selection trigger instruction), when the cursor moves to the corresponding element area, it means that the element area is the preliminary frame selection area. If a click signal is received, it means that the element area is the target frame selection area, that is, the area of ​​interest. Obtain the range data of the target frame selection area, generate an area selection instruction according to the range data, and send the area selection instruction to the first terminal device, so that the first terminal device determines the range of the area of ​​interest in the first screen image based on the range data in the area selection instruction, and performs area screen segmentation on the first screen image according to the range of the area of ​​interest to obtain the corresponding area of ​​interest screen and non-area of ​​interest screen. For example, in the frame selection mode, assuming that the cursor moves into the element area C11, the element area C11 is first highlighted to remind the user that this area is a preliminary frame selection area. At this time, if the user performs a click operation, the second terminal device receives the click signal, performs a second highlight process on the element area C11, and determines that the element area C11 is the target frame selection area, obtains the range data of the element area C11, generates a region selection instruction according to the range data of the element area C11, and sends the region selection instruction to the first terminal device, so that the first terminal device determines the range of the area of ​​interest in the first screen image based on the range data in the region selection instruction, and performs region screen segmentation on the first screen image according to the range of the area of ​​interest to obtain the corresponding area of ​​interest screen (i.e., C11) and non-area of ​​interest screen (i.e., C10, C12, and C13). As described above, by presetting the corresponding element area in the corresponding first screen image / second screen image, the user can more conveniently operate when frame selecting the area of ​​interest, and the frame selection of the area of ​​interest can be completed based on the cursor movement and click, thereby improving the user experience.

[0113] Figure 7 is another schematic diagram of frame selection based on the second screen provided in an embodiment of the present application, referring to Figure 7 , assuming that the second screen image (i.e., the first screen image) is a design page of a corresponding design software, when in the frame selection mode (which can be triggered by a frame selection trigger instruction), the preliminary frame selection area can be determined based on the frame selection operation (click plus drag operation), for example Figure 7 The area D10 in the first screen image is shown in FIG. 10. When the confirmation instruction is received, the preliminary framed area is determined to be the target framed area, that is, the area of ​​interest. The range data of the target framed area is obtained, and an area selection instruction is generated according to the range data. The area selection instruction is sent to the first terminal device, so that the first terminal device determines the range of the area of ​​interest in the first screen image based on the range data in the area selection instruction, and performs area screen segmentation on the first screen image according to the range of the area of ​​interest to obtain the corresponding area of ​​interest screen and non-area of ​​interest screen.

[0114] It should be noted that if you need to change the area of ​​interest later, you need to trigger the frame selection trigger command and enter the frame selection mode to re-select the area of ​​interest. It should be noted that the range of the area of ​​interest is the range of relative position, that is, when the second screen is zoomed, the corresponding area of ​​interest will also be zoomed.

[0115] Figure 8 is a schematic diagram of a region of interest after the screen is reduced, as provided in an embodiment of the present application. Figure 7 and Figure 8 After selecting the region of interest, the range of the region of interest is the range of the relative position. When the second screen image is reduced, the corresponding region of interest image will also be reduced accordingly. Similarly, when the second screen image is enlarged, the corresponding region of interest image will also be enlarged accordingly.

[0116] In one embodiment, assuming that the currently displayed second screen image (corresponding to the first screen image) is a vehicle design drawing, the user selects the area of ​​the corresponding key component as the target selection area, that is, the area of ​​interest, through a second terminal device (such as a mobile phone or tablet computer) in the frame selection mode. Through the aforementioned embodiment, the area image of the key component in the second screen image (that is, the area of ​​interest) can be obtained to achieve high-quality display, so that the user can clearly view the details of the key components on the second terminal device, thereby improving the user's viewing experience and improving the user's overall work efficiency in the design work on the vehicle.

[0117] S204: Send the control instruction to the first terminal device, so that the first terminal device updates the first screen image according to the control instruction.

[0118] The control instructions can be understood as instructions related to interactive control, such as click, zoom, delete, add, switch and area selection instructions. The second terminal device sends the control instruction to the first terminal device, and the first terminal device updates the first screen image based on the control instruction, and re-executes S101-S105 on the updated first screen image, so that the second terminal device can receive the coded data corresponding to the updated first screen image and display the updated second screen image. Through the interactive control of the first terminal device and the second terminal device mentioned above, the effect of controlling the first screen image corresponding to the first terminal device on the second terminal device is achieved, that is, the effect of remote desktop control is achieved, so that high-performance work, such as complex design work, can be performed on the second terminal device with relatively low performance configuration, thereby saving overall resource configuration, reducing the cost investment of the user end, and thus improving the overall user experience.

[0119] The above is used for the second terminal device, which receives the encoded data, decodes the encoded data, renders the decoded data, displays the second screen, receives the interactive signal in the process of displaying the second screen, generates the corresponding control instruction in response to the interactive signal, and sends the control instruction to the first terminal device, so that the first terminal device updates the first screen according to the control instruction. By adopting the above technical means, in the process of displaying the second screen on the second terminal device, based on the interactive control of the second screen, the interactive control of the first screen of the first terminal device can be realized, and the remote desktop service function can be realized, so that the interactive control operation with high performance configuration requirements can be realized on the second terminal device with relatively low performance configuration, thereby improving the flexibility of interactive control, thereby improving the user experience, and improving the work efficiency corresponding to the user interactive control.

[0120] Based on the above embodiments, Fig. 9 A schematic diagram of the structure of an interactive control device based on screen content provided by an embodiment of the present application. Fig. 9 The screen content-based interactive control device provided in this embodiment is used for a first terminal device, and specifically includes: an information acquisition module 21, a screen segmentation module 22, an encoding module 23, a data sending module 24 and an instruction receiving module 25.

[0121] The information acquisition module 21 is used to acquire the bandwidth information of the currently displayed first screen image and the currently connected network;

[0122] The image segmentation module 22 is used to segment the first screen image into regions to obtain an image of an area of ​​interest and an image of an area of ​​non-interest;

[0123] The encoding module 23 is used to perform a first encoding process on the picture in the region of interest, and perform a second encoding process on the picture in the region of non-interest according to the bandwidth information, so as to obtain encoding data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process;

[0124] A data sending module 24, configured to send the encoded data to a second terminal device, so that the second terminal device displays a second screen image according to the encoded data;

[0125] The instruction receiving module 25 is used to receive the control instruction generated and sent by the second terminal device during the process of displaying the second screen image, and update the first screen image according to the control instruction.

[0126] In one embodiment, the picture segmentation module 22 includes a convolution segmentation submodule, a predefined segmentation submodule, and a frame selection segmentation submodule;

[0127] A convolution segmentation submodule, used to identify the display content of the first screen image through a convolutional neural network model, and perform regional image segmentation on the first screen image according to the identification result to obtain an area of ​​interest image and a non-area of ​​interest image;

[0128] A predefined segmentation submodule, used to segment a predefined area in the window area corresponding to the target software in the first screen image into an area of ​​interest image, and segment other areas into non-area of ​​interest images;

[0129] The frame selection and segmentation submodule is used to segment the area specified by the area selection instruction sent by the second terminal device into an area of ​​interest screen, and segment other areas into non-area of ​​interest screens.

[0130] In one embodiment, the encoding module 23 includes a first encoding submodule, a content classification submodule, a screen segmentation submodule, a second encoding submodule and a data integration submodule;

[0131] A first encoding submodule, used for performing a first encoding process on the picture of the region of interest to obtain first encoded data corresponding to the picture of the region of interest;

[0132] The content classification submodule is used to identify the content of the non-interest area pictures, and classify them according to the identification results to obtain the content classification results;

[0133] The screen segmentation submodule is used to segment the non-interest area screen according to the content classification result to obtain the sub-area screen corresponding to the content classification result;

[0134] A second encoding submodule, used for transmitting the sub-region picture to a corresponding encoder, and controlling the encoder to perform a second encoding process on the sub-region picture according to the bandwidth information to obtain second encoded data;

[0135] The data integration submodule is used to integrate the first coded data and the second coded data to obtain coded data corresponding to the first screen image.

[0136] In one embodiment, the data sending module 24 includes a first comparison submodule, a third encoding submodule, a first data sending submodule, and a second data sending submodule;

[0137] A first comparison submodule is used to compare the bandwidth information with a first preset bandwidth threshold, and when the bandwidth information exceeds the first preset bandwidth threshold, send the encoded data to the second terminal device, so that the second terminal device displays a second screen image according to the encoded data;

[0138] A third encoding submodule is used to perform a third encoding process on the first screen image when the bandwidth information is below a first preset bandwidth threshold, to obtain third encoded data corresponding to the first screen image, wherein the image quality of the third encoding process is lower than the image quality of the second encoding process;

[0139] A first data sending submodule, used for sending the third coded data to the second terminal device, so that the second terminal device displays a third screen image according to the third coded data;

[0140] The second data sending submodule is used to send the first coded data to the second terminal device, so that the second terminal device refreshes and displays the corresponding area picture in the third screen picture according to the first coded data.

[0141] In one embodiment, the sub-region screen includes a first type of sub-region screen and a second type of sub-region screen, the first type of sub-region screen includes a text region screen, a line region screen and a solid color picture region screen, and the second type of sub-region screen is the remaining region screens in the non-interest region screen except the first type of sub-region screen;

[0142] The second encoding submodule includes a picture transmission unit, a first comparison unit, an encoding unit and a data integration unit;

[0143] A picture transmission unit, used for transmitting the first type of sub-region pictures to the first encoder, and transmitting the second type of sub-region pictures to the second encoder;

[0144] a first comparing unit, configured to compare the bandwidth information with a second preset bandwidth threshold, and when the bandwidth information exceeds the second preset bandwidth threshold, control the first encoder to perform fourth encoding processing on the first-type sub-region picture to obtain first sub-encoded data, and the second preset bandwidth threshold is greater than the first preset bandwidth threshold;

[0145] an encoding unit, used for controlling the second encoder to perform a fifth encoding process on the second type of sub-region picture to obtain second sub-encoded data, wherein the picture quality of the fourth encoding process is higher than the picture quality of the fifth encoding process, the picture quality of the fourth encoding process is equal to or lower than the picture quality of the first encoding process, and the picture quality of the fifth encoding process is higher than the picture quality of the third encoding process;

[0146] The data integration unit is used to integrate the first sub-coded data and the second sub-coded data to obtain the second coded data corresponding to the non-interest area picture.

[0147] In one embodiment, the first comparison unit is further used to compare the bandwidth information with the first preset bandwidth threshold and the second preset bandwidth threshold, and when the bandwidth information is below the second preset bandwidth threshold and exceeds the first preset bandwidth threshold, control the first encoder to perform fifth encoding processing on the first type of sub-region picture to obtain third sub-encoded data;

[0148] The encoding unit is further used to control the second encoder to perform fifth encoding processing on the second sub-region picture to obtain fourth sub-encoded data;

[0149] The data integration unit is further used to integrate the third sub-coded data and the fourth sub-coded data to obtain second coded data corresponding to the non-interest area picture.

[0150] The screen content-based interactive control device provided in the embodiment of the present application can be used to execute the screen content-based interactive control method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0151] Based on the above embodiments, Fig.10 This is a schematic diagram of the structure of another interactive control device based on screen content provided by an embodiment of the present application. Fig.10 The screen content-based interactive control device provided in this embodiment is used for a second terminal device, and specifically includes: a data receiving module 31, a decoding and rendering module 32, a control instruction generating module 33 and an instruction sending module 34.

[0152] The data receiving module 31 is used to receive coded data, which is obtained by the first terminal device performing a first coding process on the interested area picture in the first screen picture and performing a second coding process on the non-interested area picture in the first screen picture according to the bandwidth information;

[0153] A decoding and rendering module 32, configured to decode the encoded data, render the decoded data, and display a second screen image, the second screen image corresponding to the first screen image of the first terminal device;

[0154] A control instruction generating module 33, configured to receive an interactive signal during the display of the second screen image, and generate a control instruction in response to the interactive signal;

[0155] The instruction sending module 34 is used to send the control instruction to the first terminal device, so that the first terminal device updates the first screen image according to the control instruction.

[0156] The screen content-based interactive control device provided in the embodiment of the present application can be used to execute the screen content-based interactive control method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0157] The present application embodiment provides an interactive control device based on screen content, referring to Fig.11 The interactive control device based on screen content includes: a processor 41, a memory 42, a communication module 43, an input device 44 and an output device 45. The number of processors in the interactive control device based on screen content can be one or more, and the number of memories in the interactive control device based on screen content can be one or more. The processor, memory, communication module, input device and output device of the interactive control device based on screen content can be connected via a bus or other methods.

[0158] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the interactive control method based on screen content described in any embodiment of the present application (for example, an information acquisition module, a screen segmentation module, a coding module, a data sending module and an instruction receiving module in an interactive control device based on screen content). The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0159] The communication module 43 is used for data transmission.

[0160] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory, that is, realizes the above-mentioned interactive control method based on screen content.

[0161] The input device 44 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 45 may include a display device such as a display screen.

[0162] The screen content-based interactive control device provided above can be used to execute the screen content-based interactive control method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0163] The embodiment of the present application also provides a storage medium storing computer executable instructions, which are used to execute a screen content-based interactive control method when executed by a computer processor. The screen content-based interactive control method includes: obtaining a first screen image currently displayed and bandwidth information of a currently connected network; performing region segmentation on the first screen image to obtain an interest region image and a non-interest region image; performing a first encoding process on the interest region image, and performing a second encoding process on the non-interest region image according to the bandwidth information to obtain encoded data corresponding to the first screen image, wherein the image quality of the first encoding process is higher than the image quality of the second encoding process; sending the encoded data to a second terminal device so that the second terminal device displays the second screen image according to the encoded data; receiving a control instruction generated and sent by the second terminal device during the display of the second screen image, and updating the first screen image according to the control instruction.

[0164] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0165] Of course, the computer executable instructions of a storage medium storing computer executable instructions provided in an embodiment of the present application are not limited to the interactive control method based on screen content as described above, and can also execute related operations in the interactive control method based on screen content provided in any embodiment of the present application.

[0166] The screen content-based interactive control apparatus, storage medium, and screen content-based interactive control device provided in the above embodiments can execute the screen content-based interactive control method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the screen content-based interactive control method provided in any embodiment of the present application.

[0167] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. An interactive control method based on screen content, It is characterized in that For a first terminal device, comprising: Obtain the first screen image currently being displayed and the bandwidth information of the currently connected network; Performing regional screen segmentation on the first screen image to obtain an area of ​​interest image and an area of ​​non-interest image; Performing a first encoding process on the picture in the region of interest, and performing a second encoding process on the picture in the region of non-interest according to the bandwidth information, to obtain encoding data corresponding to the first screen picture, wherein the picture quality of the first encoding process is higher than the picture quality of the second encoding process; sending the encoded data to a second terminal device, so that the second terminal device displays a second screen image according to the encoded data; Receive a control instruction generated and sent by the second terminal device during the process of displaying the second screen image, and update the first screen image according to the control instruction.

2. The method according to claim 1, It is characterized in that The performing of region segmentation on the first screen image to obtain an area of ​​interest image and an area of ​​non-interest image comprises: Recognize the display content of the first screen image by using a convolutional neural network model, and segment the first screen image into regions according to the recognition result to obtain an image of an area of ​​interest and an image of an area of ​​non-interest; Dividing a predetermined area in the window area corresponding to the target software in the first screen image into an area-of-interest image, and dividing other areas into non-area-of-interest images; The area specified by the area selection instruction sent by the second terminal device is divided into an area of ​​interest screen, and other areas are divided into non-area of ​​interest screens.

3. The method according to claim 1, It is characterized in that The performing a first encoding process on the picture in the region of interest, and performing a second encoding process on the picture in the region of non-interest according to the bandwidth information, to obtain encoded data corresponding to the first screen picture, includes: Performing a first encoding process on the region-of-interest picture to obtain first encoded data corresponding to the region-of-interest picture; Performing content recognition on the non-interest area image, and performing classification processing according to the recognition result to obtain a content classification result; Performing region segmentation on the non-interest region picture according to the content classification result to obtain sub-region pictures corresponding to the content classification result; Transmitting the sub-region picture to a corresponding encoder, and controlling the encoder to perform a second encoding process on the sub-region picture according to the bandwidth information to obtain second encoded data; The first coded data and the second coded data are integrated to obtain coded data corresponding to the first screen image.

4. The method according to claim 3, It is characterized in that The sending of the encoded data to the second terminal device so as to display a second screen image according to the encoded data on the second terminal device comprises: comparing the bandwidth information with a first preset bandwidth threshold, and when the bandwidth information exceeds the first preset bandwidth threshold, sending the encoded data to a second terminal device, so that the second terminal device displays a second screen image according to the encoded data; When the bandwidth information is below the first preset bandwidth threshold, performing a third encoding process on the first screen image to obtain third encoded data corresponding to the first screen image, wherein the image quality of the third encoding process is lower than the image quality of the second encoding process; sending the third encoded data to the second terminal device, so that the second terminal device displays a third screen according to the third encoded data; The first coded data is sent to the second terminal device, so that the second terminal device refreshes and displays the corresponding area picture in the third screen picture according to the first coded data.

5. The method according to claim 4, It is characterized in that The sub-region pictures include a first type of sub-region pictures and a second type of sub-region pictures, the first type of sub-region pictures include a text region picture, a line region picture and a solid color picture region picture, and the second type of sub-region pictures are the remaining region pictures in the non-interest region picture except the first type of sub-region pictures; The transmitting the sub-region picture to a corresponding encoder, and controlling the encoder to perform a second encoding process on the sub-region picture according to the bandwidth information to obtain second encoded data, includes: Transmitting the first-type sub-region picture to a first encoder, and transmitting the second-type sub-region picture to a second encoder; comparing the bandwidth information with a second preset bandwidth threshold, and when the bandwidth information exceeds the second preset bandwidth threshold, controlling the first encoder to perform fourth encoding processing on the first-type sub-region picture to obtain first sub-encoded data, and the second preset bandwidth threshold is greater than the first preset bandwidth threshold; Control the second encoder to perform fifth encoding processing on the second-type sub-region picture to obtain second sub-encoded data, wherein the picture quality of the fourth encoding processing is higher than the picture quality of the fifth encoding processing, the picture quality of the fourth encoding processing is equal to or lower than the picture quality of the first encoding processing, and the picture quality of the fifth encoding processing is higher than the picture quality of the third encoding processing; The first sub-coded data and the second sub-coded data are integrated to obtain second coded data corresponding to the non-interest area picture.

6. The method according to claim 5, It is characterized in that After transmitting the first-type sub-region picture to the first encoder and transmitting the second-type sub-region picture to the second encoder, the method further comprises: comparing the bandwidth information with a first preset bandwidth threshold and a second preset bandwidth threshold, and when the bandwidth information is below the second preset bandwidth threshold and exceeds the first preset bandwidth threshold, controlling the first encoder to perform fifth encoding processing on the first-type sub-region picture to obtain third sub-encoded data; controlling the second encoder to perform fifth encoding processing on the second sub-region picture to obtain fourth sub-encoded data; The third sub-coded data and the fourth sub-coded data are integrated to obtain second coded data corresponding to the non-interest area picture.

7. An interactive control method based on screen content, It is characterized in that For the second terminal device, including: Receiving coded data, where the coded data is obtained by performing a first coding process on an area of ​​interest in a first screen image by a first terminal device and performing a second coding process on an area of ​​non-interest in the first screen image according to bandwidth information; Decoding the encoded data, rendering the decoded data, and displaying a second screen image, where the second screen image corresponds to the first screen image of the first terminal device; receiving an interactive signal during the display of the second screen image, and generating a control instruction in response to the interactive signal; The control instruction is sent to the first terminal device, so that the first terminal device updates the first screen according to the control instruction.

8. An interactive control device based on screen content, It is characterized in that For a first terminal device, comprising: An information acquisition module, used to acquire the first screen image currently displayed and the bandwidth information of the currently connected network; A picture segmentation module, used for performing regional picture segmentation on the first screen picture to obtain a picture of an area of ​​interest and a picture of a non-area of ​​interest; an encoding module, configured to perform a first encoding process on the image in the region of interest, and perform a second encoding process on the image in the region of non-interest according to the bandwidth information, to obtain encoding data corresponding to the first screen image, wherein the image quality of the first encoding process is higher than the image quality of the second encoding process; A data sending module, used for sending the coded data to a second terminal device, so that the second terminal device displays a second screen image according to the coded data; The instruction receiving module is used to receive the control instruction generated and sent by the second terminal device during the process of displaying the second screen image, and update the first screen image according to the control instruction.

9. An interactive control device based on screen content, It is characterized in that include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A storage medium storing computer executable instructions, It is characterized in that The computer executable instructions are used to perform the method according to any one of claims 1 to 7 when executed by a processor.