Host equipment control method and host equipment system based on split screen control

Abstract

The invention provides a host equipment control method and a host equipment system based on split screen control, and the method comprises the steps: host equipment creates a virtual window, generates an initial host image in the virtual window, and sends the initial host image to terminal equipment; the terminal equipment converts the initial host image according to a size relationship between the virtual window and a display area on the terminal equipment to obtain an initial terminal image; the terminal equipment responds to the operation of the user on the display area, and generates a control instruction according to the position and size relationship of the user operation; the host device generates a target host image in the virtual window according to the control instruction, and the terminal device converts the target host image according to the size relation to obtain a target terminal image and displays the target terminal image in the display area. According to the method, the actual display output of the host equipment is simulated by creating the virtual window, so that the host equipment can be remotely controlled by using the terminal equipment even if the host equipment is not provided with an entity display.

Description

Technical Field

[0001] The present application relates to the field of computer control technology, and in particular to a host device control method and a host device system based on split-screen control. Background Art

[0002] At present, the control modes of various types of host systems are mainly divided into two categories. One is direct operation through the host's own screen, and the other is control with the help of peripherals such as mouse, keyboard, touch screen, etc. In addition, some host systems also support the use of remote controls as auxiliary input devices. However, no matter which method is used, operations in different application scenarios often require users to return to the vicinity of the host to operate the mouse or keyboard, which greatly limits the user's freedom and flexibility, resulting in a limited range of user operations and inability to achieve flexible long-distance control. In addition, in actual applications, the remote control is easy to lose and its functions are relatively single, and it is difficult to meet complex and changing application requirements. Therefore, the existing host control mode has problems such as difficult operation and peripheral limitations in many application scenarios, which makes it impossible for users to control the host system anytime and anywhere, thereby reducing the overall user experience. Summary of the invention

[0003] The purpose of this application is to provide a host device control method and a host device system based on split-screen control in response to the above-mentioned deficiencies in the prior art, so as to solve the problems in the prior art that the machine control mode is difficult to operate and has peripheral limitations in many application scenarios, resulting in the user being unable to control the host system anytime and anywhere, thereby reducing the overall user experience.

[0004] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0005] In a first aspect, an embodiment of the present application provides a host device control method, which is applied to a host device system based on split-screen control, the host device system comprising: a host device and a terminal device; the method comprising:

[0006] The host device establishes a communication connection with the terminal device, and the host device creates a virtual window;

[0007] The host device generates an initial host image in the virtual window, and sends the initial host image to the terminal device;

[0008] The terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area;

[0009] The terminal device responds to the user's operation on the display area, generates a control instruction according to the position of the user operation and the size relationship, and sends the control instruction to the host device;

[0010] The host device generates a target host image in the virtual window according to the control instruction, and sends the target host image to the terminal device. The terminal device converts the target host image according to the size relationship to obtain a target terminal image, and displays the target terminal image in the display area.

[0011] As a possible implementation, the terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area, including:

[0012] Determine an image scaling ratio according to the size relationship between the virtual window and the display area;

[0013] Scale the initial host image based on the image scaling ratio to obtain the initial terminal image;

[0014] Map the initial terminal image to the display area to display the initial terminal image in the display area.

[0015] As a possible implementation, the mapping of the initial terminal image to the display area to display the initial terminal image in the display area includes:

[0016] Map each pixel point in the scaled host image from the original position to a new position in the display area, and adjust the pixel value at the new position to the pixel value of the pixel point.

[0017] As a possible implementation, the terminal device responds to the user's operation on the display area, generates a control instruction according to the position of the user operation and the size relationship, including:

[0018] Perform coordinate conversion on the position of the user operation based on the size relationship to determine the target position of the user operation in the virtual window;

[0019] Generate the control instruction according to the target position of the user operation in the virtual window, the user operation, and the execution time of the user operation.

[0020] As a possible implementation, the performing coordinate conversion on the position of the user operation based on the size relationship to determine the target position of the user operation in the virtual window includes:

[0021] According to the size relationship, determine the proportional information of the position of the user operation in the virtual window, where the proportional information is used to indicate the percentage of the position of the user operation in the virtual window;

[0022] Based on the proportional information, perform coordinate transformation on the position of the user operation to determine the target position.

[0023] As a possible implementation, the host device generates a target host image in the virtual window according to the control instruction, including:

[0024] Parse the control instruction to obtain the target position of the user operation in the virtual window and the user operation event information;

[0025] Update the image content in the virtual window according to the target position and the user operation event information, and capture the update status of the virtual window in real time;

[0026] Encode the captured update status of the virtual window to generate the target host image.

[0027] As a possible implementation, the host device establishes a communication connection with the terminal device, including:

[0028] The terminal device responds to the input operation of the user on the software setting interface, and sends the network information of the local area network to the host device, where the network information includes the network name and the network password;

[0029] The host device listens to the network information, connects to the local area network based on the network information, and sends a broadcast message to the terminal device, where the broadcast message includes the device name and the device unique identification code;

[0030] The terminal device listens to the broadcast message and establishes a communication connection with the host device based on the broadcast message.

[0031] As a possible implementation, the terminal device listens to the broadcast message and establishes a communication connection with the host device, including:

[0032] In response to the user's click operation on the device acquisition button, listen for the existence of the broadcast message. If so, establish a communication connection with the host device based on the device name and the device unique identification code.

[0033] As a possible implementation, it further includes:

[0034] The terminal device responds to the user's operation of selecting a target application, and sends the application information of the target application to the host device, where the application information includes the application name and the download link of the target application;

[0035] The host device receives the application information, and downloads and installs the target application based on the download link.

[0036] In a second aspect, an embodiment of the present application provides a host device system based on split-screen control, where the host device system includes: a host device and a terminal device;

[0037] The host device is configured to execute the method steps executed by the host device in any one of the host device control methods in the first aspect above;

[0038] The terminal device is configured to execute the method steps executed by the terminal device in any one of the host device control methods in the first aspect above.

[0039] According to the host device control method and the host device system based on split-screen control in the embodiments of the present application, the host device establishes a communication connection with the terminal device, the host device creates a virtual window, generates an initial host image in the virtual window, and sends the initial host image to the terminal device. The terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area. The terminal device responds to the user's operation on the display area, generates a control instruction according to the position and size relationship of the user's operation, and sends the control instruction to the host device. The host device generates a target host image in the virtual window according to the control instruction, and sends the target host image to the terminal device. The terminal device converts the target host image according to the size relationship to obtain a target terminal image, and displays the target terminal image in the display area. According to the embodiments of the present application, by creating a virtual window to simulate the actual display output of the host device, the host device can generate content that can be remotely viewed and interacted with even without a physical display. Moreover, by adjusting the host image according to the size relationship of the display area of the terminal device, it is ensured that no matter the screen size of the terminal device, the content can be displayed in the best proportion, enhancing the adaptability of the interface and the user's visual experience. In addition, by establishing a wireless connection between the terminal device and the host device, it is allowed to perform data transmission between the host device and the terminal device without physical connection. Every operation of the user on the terminal device will be quickly reflected on the host device and the display content will be updated immediately, breaking the limitation of the traditional need to use cables or other hardware interfaces, and using a portable terminal device such as a smart phone as a controller, the user can operate the host device anywhere and at any time, greatly improving the flexibility and convenience of use. Description of the Drawings

[0040] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following accompanying drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.

[0041] Figure 1 Shows a schematic diagram of the architecture of a host device system based on split-screen control provided by an embodiment of the present application;

[0042] Figure 2 Shows a schematic diagram of the structure of a host device provided by an embodiment of the present application;

[0043] Figure 3 Shows a schematic flowchart of a method for controlling a host device provided by an embodiment of the present application;

[0044] Figure 4 Shows a schematic diagram of a display area and a virtual interface provided by an embodiment of the present application;

[0045] Figure 5 Shows a schematic flowchart of a method for determining an initial terminal image provided by an embodiment of the present application;

[0046] Figure 6 Shows a schematic flowchart of a method for generating a control instruction provided by an embodiment of the present application;

[0047] Figure 7 Shows a schematic diagram of a coordinate transformation provided by an embodiment of the present application;

[0048] Figure 8 Shows a schematic flowchart of a method for generating a target host image provided by an embodiment of the present application;

[0049] Figure 9 Shows a schematic flowchart of a method for establishing a communication connection provided by an embodiment of the present application;

[0050] Figure 10 Shows a schematic diagram of an application download interface provided by an embodiment of the present application. Detailed implementation manners

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. It should be understood that the accompanying drawings in this application are only for the purposes of illustration and description, and are not used to limit the protection scope of this application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of this application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without a logical context relationship may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of this application.

[0052] In addition, the described embodiments are only some embodiments of this application, rather than all of the embodiments. The components of the embodiments of this application usually described and illustrated in the accompanying drawings here may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application claimed, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts fall within the protection scope of this application.

[0053] It should be noted that the term "including" will be used in the embodiments of this application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0054] In many application scenarios, users often need to externally connect the host device to other display devices, such as TVs, monitors, or other special-shaped screens. And if they need to operate the host device, they need to constantly go near the host device to operate the mouse or touch screen. These inconvenient operations seriously affect the user experience.

[0055] To address the above problems, this application provides a host device control method and a host device system based on split-screen control. The host device itself has no display screen, but creates a virtual interface. Through a terminal device, such as a smart phone, the host device can be remotely controlled, and it supports users to remotely control the host device within the same local area network or across the Internet, and use the screen of the terminal device as the output display device of the host device, thereby achieving the technical effect of "screens are everywhere". This not only greatly increases the utilization rate of various screens, but also makes the experience operation of all screens consistent with the terminal device.

[0056] The following will separately elaborate on the host device system based on split-screen control and the host device control method provided by this application.

[0057] Figure 1 It shows a schematic architecture diagram of a host device system based on split-screen control provided by an embodiment of the present application. Refer to Figure 1 As shown, the host device system includes a host device and a terminal device. The host device itself has no input / output peripherals, no sensors such as gyroscopes, and only retains a power module, a main control module, a storage module, a communication module, and a built-in operating system. All controls of the host device are input by the terminal device.

[0058] Optionally, refer to Figure 2 As shown, the host device includes a power module, a main control module, a storage module, and a communication module. Among them, the power module is used to provide power support, the main control module is used to process and control the operations of the entire host device, the storage module is used to store data and programs, and the communication module includes various communication methods, such as wired network, WiFi, Bluetooth, etc., for communicating with external devices (such as terminal devices). It should be particularly emphasized that the host device of the present application itself does not have traditional input / output devices, such as keyboards, mice, monitors, etc., nor does it have sensors such as gyroscopes.

[0059] Optionally, refer to Figure 2 As shown, the power interface is used to connect to the power supply, the HDMI interface can be used to connect to external devices, the network port is used for wired network connection, the mode button is used for user operation and switching the working mode of the host device, and the USB interface is used for data transmission, etc. In addition, an operating system OS runs inside the host device, enabling the host device to run software applications independently and interact with other devices through the communication module.

[0060] Optionally, the user controls the host device through a terminal device, such as a mobile phone carried by the user himself / herself. The control operations include but are not limited to software installation, uninstallation, opening, operation, closing, etc. The method of controlling the host device provided by the present application enables the user to perform agile control of the host device without interrupting his / her own activities.

[0061] Based on this, the host device provided by the present application achieves high portability and flexibility by streamlining the hardware configuration. The host device relies on a terminal device, such as the user's mobile phone, to perform all necessary control operations, thereby achieving agile control without interrupting the user's own activities. This control method is particularly suitable for scenarios that require frequent movement or use in different environments. And through this control method, the host device not only simplifies its own hardware structure, but also enhances the interactivity between the host device and the user's existing devices (such as mobile phones), improving the overall user experience.

[0062] The following combines the above Figure 1Based on the content described in the host device system based on split - screen control shown, the host device control method provided in the embodiments of the present application will be described in detail.

[0063] Figure 3 The flowchart of a host device control method provided in the embodiments of the present application is shown. Referring to Figure 3 as shown, the execution subject of this method is the host device and the terminal device in the host device system, and specifically includes the following steps:

[0064] S301. The host device establishes a communication connection with the terminal device.

[0065] Optionally, establishing a communication connection between the host device and the terminal device is to ensure that data can be transmitted between the host device and the terminal device, and to ensure that the terminal device can remotely control the host device by establishing a two - way data transmission channel. Specifically, the communication connection between the host device and the terminal device can be established through network protocols (such as TCP / IP, WebSocket) or local connections (such as USB, Bluetooth), etc.

[0066] S302. The host device creates a virtual window, generates an initial host image in the virtual window, and sends the initial host image to the terminal device.

[0067] Optionally, the virtual window is a logical display area on the host device, which can represent an application window, a screen area, or an independently rendered canvas. Since the host device itself does not have input / output peripherals and a display screen, in the present application, a virtual window is created inside the host device. This virtual window can be regarded as simulating an actual display, providing a space for the application programs on the host device to display their interfaces, so as to transmit their screen contents to the terminal device.

[0068] Optionally, application programs on the host device can be rendered and displayed in the virtual window to generate an initial host image in the virtual window. The host device encodes the initial host image and sends the initial host image to the terminal device in the form of image encoding and video stream through the previously established communication connection.

[0069] S303. The terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area.

[0070] Optionally, the size relationship is used to reflect the differences in resolution and aspect ratio between the virtual window and the display area. Referring to Figure 4As shown, the resolution and screen size of the display area on the terminal device are different from those of the virtual window created by the host device. To correctly display the virtual window of the host device in the display area of the terminal device, the resolution and screen size of the virtual window and the display area need to be adapted.

[0071] Optionally, after receiving the initial host image from the host device, the terminal device adjusts the proportion of the initial host image based on the actual size and resolution of the screen corresponding to its display area, and adjusts the size of the initial host image proportionally. If necessary, cropping or padding can be performed to make the initial host image fit the display area of the terminal device, thereby obtaining the initial terminal image and displaying the initial terminal image in the display area of the terminal device.

[0072] S304. The terminal device responds to the user's operation on the display area, generates a control instruction according to the position and size relationship of the user operation, and sends the control instruction to the host device.

[0073] Optionally, the user can interact with the application program of the host device through the terminal device. Specifically, the user performs operations such as touching or clicking on the display area of the terminal device. The terminal device identifies the input position of the user operation, calculates the coordinate information corresponding to the input position of the user operation on the host device according to the size relationship between the virtual window and the display area on the terminal device, and then generates a corresponding control instruction based on the coordinate information. The types of control instructions include but are not limited to mouse events (move, click, scroll wheel), keyboard events (key input), gestures or multi-touch instructions. Then, the terminal device sends the control instruction back to the host device based on the previously established communication connection.

[0074] Optionally, the user operations include but are not limited to input events such as touching, clicking, and dragging. Coordinate conversion is performed based on the position and size relationship of the user operation to map the user operation position on the terminal device to the coordinate system of the virtual window. For example, the click position on the display area of the terminal device is (240, 135), and the image scaling ratio determined according to the size relationship between the virtual window and the display area on the terminal device is 2:1. Then the click position (240, 135) corresponds to the position (480, 270) in the virtual window.

[0075] S305. The host device generates a target host image in the virtual window according to the control instruction and sends the target host image to the terminal device.

[0076] Optionally, when the user generates a user operation on the terminal device, it is necessary to reflect the impact of the user operation on the content running on the host device and update the display content of the virtual window in the host device. After receiving the control instruction, the host device needs to update the virtual window state according to the control instruction. Specifically, it performs corresponding operations according to the instruction content of the control instruction, such as clicking a certain button, moving the mouse cursor, opening an application, or scrolling the page. These operations will cause changes in the interface in the virtual window, generating a new target host image. The new target host image is also encoded and sent to the terminal device in the form of a video stream through the communication connection.

[0077] It should be noted that when sending the target host image to the terminal device, only the difference frame (incremental update) can be sent, or the complete image can be sent. The sending method specifically depends on the performance requirements and is not specifically limited here.

[0078] S306. The terminal device converts the target host image according to the size relationship to obtain a target terminal image and displays the target terminal image in the display area.

[0079] Optionally, in order to ensure that the changes on the virtual interface of the host device can be accurately presented on the display area of the terminal device, after receiving the target host image from the host device, the terminal device converts the target host image again according to the size relationship, adjusts the target host image to a target terminal image adapted to the display area of the terminal device, and displays the target terminal image on the display area.

[0080] Based on this, according to the host device control method provided by the embodiments of the present application, by creating a virtual window to simulate the actual display output of the host device, the host device can generate content that can be remotely viewed and interacted with even without a physical display. Moreover, by adjusting the host image according to the size relationship of the display area of the terminal device, it is ensured that no matter the screen size of the terminal device, the content can be displayed in the best proportion, enhancing the adaptability of the interface and the user's visual experience. In addition, by establishing a wireless connection between the terminal device and the host device, it is allowed to perform data transmission between the host device and the terminal device without a physical connection. Every operation of the user on the terminal device will be quickly reflected on the host device and the display content will be updated immediately, breaking the limitation of the traditional need to use cables or other hardware interfaces. And by using a portable terminal device such as a smart phone as a controller, no matter how far the physical distance is between the host device and the terminal device, the user can use the terminal device to browse and control the applications on the host device just like directly operating the host device, realizing remote control of the host device, greatly improving the flexibility and convenience of use.

[0081] Figure 5The figure shows a schematic flowchart of a method for determining an initial terminal image provided by an embodiment of the present application. Refer to Figure 5 As shown, in step S303, the terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain the initial terminal image, and displays the initial terminal image in the display area. The specific steps are as follows:

[0082] S501. Determine the image scaling ratio according to the size relationship between the virtual window and the display area.

[0083] Optionally, the image scaling ratio is an equal ratio scaling ratio to ensure that the content of the virtual window on the host device can be adapted to the display area of the terminal device, ensuring that the display effect is not distorted and making full use of the screen space of the display area.

[0084] Specifically, the terminal device obtains the size information of the virtual window, such as width and height, determines the aspect ratio of the virtual window, compares the aspect ratio of the display area on the terminal device with the aspect ratio of the virtual window, and then calculates the image scaling ratio to be applied based on the selected scaling reference to ensure that the initial host image can be completely displayed in the display area of the terminal device. Among them, if the virtual window is wider, the image scaling ratio is calculated with the width of the display area of the terminal device as the limit; if the virtual window is higher, the image scaling ratio is calculated with the height of the display area of the terminal device as the limit.

[0085] S502. Scale the initial host image based on the image scaling ratio to obtain the initial terminal image.

[0086] Optionally, scaling the initial host image based on the image scaling ratio actually adjusts the size of the initial host image according to the image scaling ratio to make the initial host image fit the display area of the terminal device. The scaled image is the initial terminal image, and the resolution and size of the initial terminal image have been adjusted to the state most suitable for the display area on the terminal device.

[0087] Optionally, an image processing algorithm can be used to perform the scaling operation on the original initial host image. Among them, the image processing algorithm includes but is not limited to nearest neighbor interpolation, bilinear interpolation or more advanced algorithms. Exemplarily, taking the nearest neighbor interpolation algorithm as an example, the nearest neighbor interpolation algorithm is a resampling method. For each pixel point in the initial host image, by determining the closest corresponding point of each pixel point in the source image and using the color value of that corresponding point. Specifically, for each pixel position in the initial host image, the corresponding source image position is found through inverse mapping, rounded to the closest integer coordinate, and then the pixel color value at that integer coordinate position is assigned to the terminal image in the display area, that is, the initial terminal image is obtained.

[0088] S503. Map the initial terminal image to the display area to display the initial terminal image in the display area.

[0089] Optionally, mapping the initial terminal image to the display area means accurately presenting the scaled initial terminal image on the display area of the terminal device. Exemplarily, mapping the initial terminal image to the display area specifically maps each pixel point in the scaled host image from its original position to a new position in the display area, and adjusts the pixel value at the new position to the pixel value of the pixel point. Since simply calculating the new position of each pixel based on the image scaling ratio may cause some problems, such as when the image is enlarged, there may be no directly corresponding original pixel point for the new pixel position, or when the image is reduced, multiple original pixels need to be merged into one new pixel. Therefore, resampling techniques are usually used to determine the pixel value at the new position, and once the optimal value of the pixel at the new position is determined, this value is assigned to the pixel at the corresponding position in the new image.

[0090] It should be noted that the process of mapping each pixel point in the scaled host image from its original position to a new position in the display area and adjusting the pixel value at the new position to the pixel value of the pixel point is essentially an image resampling process. This process not only includes the transformation of pixel positions, but also includes appropriate adjustment of pixel values according to different algorithms to ensure the quality of the final display effect, thereby ensuring that even after the scaling operation, the image can still maintain high clarity and visual effects.

[0091] Optionally, during the process of positioning and mapping the scaled initial terminal image into the display area of the terminal device, some additional adjustments may also be involved. For example, if the size of the scaled initial terminal image is smaller than the display area, it may be necessary to center the display or add a border. If the initial terminal image is larger than the display area, it may be necessary to further crop it. Finally, the processed image data is transmitted to the display controller so that it can be correctly rendered on the display area of the terminal device, that is, the initial terminal image is displayed in the display area.

[0092] Based on this, the terminal device can effectively adapt to the running content from host devices with different resolutions and sizes, ensuring that users can obtain a good visual experience regardless of the type of terminal device used and can conveniently interact with the host device.

[0093] Figure 6 The flowchart shows a control instruction generation method provided by an embodiment of the present application. Refer to Figure 6 As shown, in step S304 above, the terminal device responds to the user's operation on the display area and generates a control instruction according to the position and size relationship of the user's operation, which specifically includes the following steps:

[0094] S601. Perform coordinate transformation on the position of the user operation based on the size relationship to determine the target position of the user operation in the virtual window.

[0095] Optionally, performing coordinate transformation on the position of the user operation based on the size relationship can ensure that the operation of the user on the terminal device can be accurately mapped to the virtual window of the host device, so that the operation position corresponds to the interface elements of the host device. Specifically, obtain the operation position of the user in the display area of the terminal device (such as the X and Y coordinates of the touch point), and then use the previously determined size relationship between the virtual window and the display area of the terminal device to calculate the actual coordinates of the operation position relative to the virtual window of the host device, which can be determined specifically through proportional scaling and translation transformation, that is, determine the target position of the user operation in the virtual window to ensure that the user operation position can accurately correspond to the position on the host device interface.

[0096] Optionally, the proportional information of the position of the user operation in the virtual window can be determined according to the size relationship, and the position of the user operation is coordinate-transformed based on the proportional information to determine the target position. Among them, the proportional information is used to indicate the percentage of the position of the user operation in the virtual window.

[0097] Exemplarily, referring to Figure 7 as shown, Figure 7 In the figure, the left interface shows the display area on the terminal device, and the right interface shows the virtual window created by the main control device. Assume that the coordinates of the user's touch operation on the display area are (50, 25), and the resolution of the display area is 200×100, and the resolution of the virtual window is 1920×1080. On this basis, it is necessary to convert the operation position of the user on the display area into the actual position on the virtual window. First, calculate the proportional information of the user operation position on the display area, that is, the percentage. Specifically, the horizontal ratio is 50 / 200 = 1 / 4, and the vertical ratio is 25 / 100 = 1 / 4. Therefore, the proportional information of the user operation position on the display area is (1 / 4, 1 / 4). Further, use the above-determined proportional information to calculate the actual position of the user operation on the virtual window. Specifically, the actual coordinate in the horizontal direction is 1920×1 / 4 = 480, and the actual coordinate in the vertical direction is 1080×1 / 4 = 270. Thus, the target position of the user operation on the virtual window can be determined as (480, 270).

[0098] S602. Generate a control instruction according to the target position of the user operation in the virtual window, the user operation, and the execution time of the user operation.

[0099] Optionally, the target position refers to the coordinates of the user operation in the virtual window. The user operation reflects the operation type, which is also the type of user behavior, such as clicking, dragging, keyboard input, etc. The execution time is the timestamp when the user operation occurs, and this timestamp can ensure that the host device processes each operation in a continuous operation event in the order of occurrence of the operations.

[0100] Optionally, the control instruction contains sufficient information such as the user operation, position coordinates, and operation execution time for the host device to understand and execute corresponding actions. In response to the user operation, the terminal device identifies the user operation type, that is, identifies which specific type of interaction operation the user has performed, such as clicking, long pressing, swiping, etc. Then, according to the target position of the user operation in the virtual window and the operation type, a specific control instruction is created. For example, if the user operation is a click operation, the instruction may contain the exact coordinates where the click occurred. If it is a drag operation, in addition to the start and end coordinates, it may also include the moving speed or trajectory. Finally, the above instruction information is packaged into a message or command in a specific format to obtain the control instruction, which is sent to the host device through the established communication connection. It should be noted that the control instruction usually follows a certain predefined protocol to ensure that the host device can correctly parse and respond.

[0101] Based on this, the terminal device can effectively convert the user's physical input into meaningful control signals for the host device, thereby realizing remote control of the host device. This not only improves the user experience but also allows the user to interact with the terminal device as naturally as directly operating the host.

[0102] Figure 8 The flowchart of a target host image generation method provided by an embodiment of the present application is shown. Refer to Figure 8 As shown, in step S305 above, the host device generates a target host image in the virtual window according to the control instruction, which specifically includes the following steps:

[0103] S801. Parse the control instruction to obtain the target position of the user operation in the virtual window and the user operation event information.

[0104] Exemplarily, parse the control instruction to extract the target position of the user operation in the virtual window and user operation event information such as the execution time and execution event, and convert the control instruction sent by the terminal device into an operation command executable by the host device, so that the host device can understand the user's intention and make corresponding responses. For example, if the control instruction indicates that the user has performed a click operation at a specific coordinate, the host device needs to identify the click coordinate of the click operation and the action of the click operation.

[0105] Specifically, the original data corresponding to the control instruction can be converted into a structured instruction according to a predefined communication protocol, and the data integrity is verified, and the legality of the fields is checked. For example, whether the coordinates of the target position exceed the range of the virtual window, whether the event type is supported, etc. Then, key parameters such as the target position, operation type, and timestamp are extracted for operation mapping, and the control instruction is converted into an operation event executable by the host device.

[0106] S802. Update the image content in the virtual window according to the target position and the user operation event information, and capture the update status of the virtual window in real time.

[0107] Exemplarily, according to the parsed target position and operation event information, the host device updates the content in the virtual window. For example, if the user clicks a button, then the state of this button should be updated to "clicked", and after the update is completed, the host device needs to capture the state change of the virtual window in real time, which means that any visual change caused by the user operation needs to be recorded for subsequent generation of new image frames.

[0108] Exemplarily, the update status of the virtual window can be captured in real time through full-frame capture or incremental capture. Full-frame capture means periodically intercepting the entire picture of the virtual window, and incremental capture means only capturing the changed partial area. Obviously, the data transmission volume of incremental capture is less than that of full-frame capture.

[0109] S803. Encode the captured update status of the virtual window to generate a target host image.

[0110] Exemplarily, for the captured update status of the virtual window, the host device uses encoding technology to encode the update status, compresses the captured virtual window picture into an efficiently transmissible encoded format, and the encoded data is the target host image. The target host image contains the latest interface state and can be efficiently transmitted to the terminal device.

[0111] Based on this, the host device can dynamically update its display content according to the operations of remote users and timely feedback these changes to the users. Thus, users can see almost instant responses on the terminal device, just as smooth and natural as directly operating the host device, which not only improves the interactivity and practicality of the host control system, but also greatly enhances the user experience.

[0112] Figure 9 The flowchart of a communication connection establishment method provided by an embodiment of the present application is shown. Refer to Figure 9 As shown, the above step S301 of the host device establishing a communication connection with the terminal device specifically includes the following steps:

[0113] S901. The terminal device responds to the user's input operation on the software settings interface and sends the network information of the local area network to the host device. Among them, the network information includes the network name and the network password.

[0114] Exemplarily, the user connects the terminal device to the home local area network, downloads and installs the host control software. During the first configuration, if the host device is not connected to the wired network and is currently in the hotspot mode AP, the user needs to manually enter the network information of the home local area network in the host control software on the terminal device. After the input is completed, the host control software will encapsulate this network information into a UDP broadcast message and send it out. This broadcast message contains the network name and the network password of the home local area network. In this way, the host device can access the local area network where the terminal device is located, so that the two can be connected to the same home local area network and can communicate directly.

[0115] S902. The host device listens for the network information, connects to the local area network based on the network information, and sends a broadcast message to the terminal device. Among them, the broadcast message includes the device name and the device unique identification code.

[0116] Optionally, the host device is default in the client mode STA, but if no valid network connection is detected, it can switch to the AP mode by pressing the Wi-Fi mode button to receive data from the terminal device. When the host is in the AP mode, it will listen for the UDP broadcast message from the terminal device. Once it receives the broadcast message containing the home local area network information, the host device will use this broadcast message to try to connect to the specified home local area network. And after successfully accessing the home local area network, the host device will send one or more broadcast messages to the terminal devices in the local area network. This broadcast message usually includes the device name and the device unique identification code of the host device, which is used to identify itself and indicate its online status and the status of being ready to accept connections to the possible control software.

[0117] S903. The terminal device listens for the broadcast message and establishes a communication connection with the host device based on the broadcast message.

[0118] Optionally, after the host control software on the terminal device connects to the home local area network, it will enter the listening state and wait to receive the broadcast message from the host device. When the terminal device receives the broadcast message sent by the host device, it can extract the relevant information of the host device from it, such as the device name and the device unique identification code. Then the host control software can select the corresponding host device and initiate a request to establish a TCP connection. Once the TCP connection is successfully established, data exchange can start between the terminal device and the host device, such as transmitting control instructions or image streams, etc.

[0119] Optionally, in response to a user's click operation on the device acquisition button, listen for the existence of broadcast messages. If so, establish a communication connection with the host device based on the device name and the device unique identification code.

[0120] Exemplarily, a device acquisition button is provided on the host control software. When the user clicks the device acquisition button, the terminal device will attempt to establish a communication connection with the host device. Specifically, the user opens the host control software on the terminal device and clicks the device acquisition button on the interface. When the click action is captured, a signal for listening to broadcast messages within the local area network is started, and the terminal device enters the listening mode, waiting to receive possible broadcast messages within the local area network. After the host device successfully accesses the local area network, it will send a broadcast message containing its device name and unique identification code. The host control software continuously listens to the broadcast messages in the local area network until a qualified message is received.

[0121] Exemplarily, once the terminal device listens to a broadcast message from the host device, it will extract the device name and the device unique identification code from the broadcast message and actively initiate a TCP connection request to the host device. During this process, some protocols or authentication steps may be involved to ensure the security and reliability of the connection. After successfully establishing the TCP connection, the terminal device can start sending control instructions to the host device or receiving data streams from the host device, thereby realizing effective control of the host device.

[0122] Based on this, from the moment the user inputs the home local area network information until the terminal device successfully establishes a communication connection with the host device, the whole process ensures that even in the case of the first configuration or a change in the network environment, the host device can be conveniently and quickly connected to the established home local area network and interconnected with the host control software.

[0123] As a possible implementation manner, the method further includes: the terminal device, in response to the user's operation of selecting a target application, sends the application information of the target application to the host device, where the application information includes the application name and the download link of the target application; the host device receives the application information and downloads and installs the target application based on the download link.

[0124] Exemplarily, referring to Figure 10 As shown, an application store is built into the host control software. The user can open the application store and browse the available application list. When the user finds the target application to be installed, such as Application A, clicks Application A, the host control software captures the click operation, obtains the application name and the download link of Application A, and uses the established TCP connection to send an information packet containing the application name and the download link to the host device.

[0125] Exemplarily, the host device monitors the data stream from the terminal device, identifies that this is a request for installing a new application, extracts the application name and download link from it, and will pop up a confirmation window asking the user whether to really download and install this application. If the user confirms the installation request, the host device will start downloading the application file using the extracted download link. After the download is completed, the host device automatically starts the installation program and completes the installation of the application according to the standard process.

[0126] Based on this, the user only needs to perform a simple click operation through the control software on the terminal device to trigger a series of automated processes, and finally realize the update or expansion of the application environment of the host device.

[0127] As a possible implementation, the method further includes: the terminal device responds to the user's click operation on the screen-off control on the host control software, controls the host device to enter the sleep mode, and disconnects the wireless connection with the host device.

[0128] Exemplarily, a screen-off control is provided on the host control software. When the user clicks on the screen-off control, after the host control software captures the click event, it will send a control instruction containing a request to enter the sleep mode to the host device. After receiving the control instruction, the host device enters the sleep mode, stops screen capture, pauses video stream transmission, etc., and disconnects the communication connection between the terminal device and the host device.

[0129] In addition, during the operation of the host device without additional operations, the terminal device can switch the host control software to the background, pause the video encoding and transmission of the virtual interface of the host device, saving the performance consumption of the host device and the terminal device. When the host control software is switched to the foreground, the video encoding and stream transmission of the virtual interface are automatically restored for the convenience of the user to operate.

[0130] Based on this, in the embodiment of the present application, when the host device is not needed, the host device can be quickly placed in the low-power state through the terminal device, which not only ensures that all relevant resources are properly released, but also significantly improves the user experience.

[0131] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the method embodiments, and will not be elaborated herein. In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules is only a logical functional division, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0132] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0133] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application.

Claims

1. A host device control method, characterized in that: Applicable to a host device system based on split-screen control, the host device system comprising: a host device and a terminal device; The method comprises: The host device establishes a communication connection with the terminal device, and the host device creates a virtual window; The host device generates an initial host image in the virtual window, and sends the initial host image to the terminal device; The terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area; The terminal device responds to the user's operation on the display area, generates a control instruction according to the position of the user's operation and the size relationship, and sends the control instruction to the host device; The host device generates a target host image in the virtual window according to the control instruction, and sends the target host image to the terminal device. The terminal device converts the target host image according to the size relationship to obtain a target terminal image, and displays the target terminal image in the display area.

2. The method according to claim 1, characterized in that: The terminal device converts the initial host image according to the size relationship between the virtual window and the display area on the terminal device to obtain an initial terminal image, and displays the initial terminal image in the display area, including: Determining an image scaling ratio according to a size relationship between the virtual window and the display area; Scaling the initial host image based on the image scaling ratio to obtain the initial terminal image; The initial terminal image is mapped to the display area to display the initial terminal image in the display area.

3. The method according to claim 2, characterized in that The mapping the initial terminal image to the display area to display the initial terminal image in the display area includes: Each pixel point in the scaled host image is mapped from an original position to a new position in the display area, and the pixel value at the new position is adjusted to the pixel value of the pixel point.

4. The method according to claim 1, characterized in that: The terminal device generates a control instruction in response to a user operation on the display area according to a position of the user operation and the size relationship, including: Performing coordinate transformation on the position of the user operation based on the size relationship to determine a target position of the position of the user operation in the virtual window; The control instruction is generated according to a target position of the position of the user operation in the virtual window, the user operation, and the execution time of the user operation.

5. The method according to claim 4, characterized in that The performing coordinate transformation on the position of the user operation based on the size relationship to determine the target position of the position of the user operation in the virtual window includes: Determine, according to the size relationship, ratio information of the position operated by the user in the virtual window, wherein the ratio information is used to indicate the position percentage of the position operated by the user in the virtual window; The coordinates of the position operated by the user are transformed based on the scale information to determine the target position.

6. The method according to claim 1, characterized in that The host device generates a target host image in the virtual window according to the control instruction, including: Parsing the control instruction to obtain the target position of the user operation position in the virtual window and user operation event information; updating the image content in the virtual window according to the target position and the user operation event information, and capturing the update status of the virtual window in real time; The captured update state of the virtual window is encoded to generate the target host image.

7. The method according to claim 1, characterized in that The host device establishes a communication connection with the terminal device, including: The terminal device responds to the user's input operation on the software setting interface, and sends the network information of the local area network to the host device, wherein the network information includes a network name and a network password; The host device monitors the network information, connects to the local area network based on the network information, and sends a broadcast message to the terminal device, wherein the broadcast message includes a device name and a unique device identification code; The terminal device monitors the broadcast message and establishes a communication connection with the host device based on the broadcast message.

8. The method according to claim 7, characterized in that The terminal device monitors the broadcast message and establishes a communication connection with the host device based on the broadcast message, including: In response to a user clicking operation on a device acquisition button, monitoring whether the broadcast message exists, and if so, establishing a communication connection with the host device based on the device name and the device unique identification code.

9. The method according to claim 1, characterized in that: Also includes: In response to the user selecting the target application, the terminal device sends application information of the target application to the host device, where the application information includes an application name and a download link of the target application; The host device receives the application information, and downloads and installs the target application based on the download link.

10. A host device system based on split-screen control, characterized in that: The host device system comprises: a host device and a terminal device; The host device is used to execute the method steps performed by the host device in the host device control method described in any one of claims 1 to 9; The terminal device is used to execute the method steps executed by the terminal device in the host device control method described in any one of claims 1-9.

Images