Screen display control method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of screen display, in particular to a screen display control method and device, computer equipment and a storage medium, and the method comprises the steps that a display device and a terminal device establish wireless connection and negotiate to determine a protocol channel to receive screen data; the display device performs display processing on the received screen data, determines a picture conversion parameter, maps a touch coordinate based on the picture conversion parameter after obtaining user touch data, generates a control instruction and issues the control instruction to the terminal device; and the terminal equipment feeds back updated screen data after executing the instruction, and the display equipment updates display in real time. According to the method and the device, natural interaction can be realized on the premise of ensuring compatibility and low delay, and the user operation experience in a wireless screen projection scene is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of screen display technology, and in particular to a screen display control method, device, computer equipment and storage medium. Background Art

[0002] With the widespread adoption of mobile terminals and touch-screen display devices, users are increasingly demanding content projection and interactive control in multi-device environments. In particular, in scenarios such as education, demonstrations, and remote control, using large-screen displays to display mobile terminal images and enabling direct interactive control has become an important form of human-computer interaction.

[0003] Existing screen display control technologies typically use wireless projection solutions based on protocols such as Miracast and AirPlay to project the screen content of terminal devices such as smartphones or tablets onto display devices. Some protocols, such as Miracast, define a reverse channel for user input to support reverse control of terminal devices by display devices. However, in actual applications, due to the poor compatibility of standard protocols between different devices, the reverse control function is unavailable or the experience is poor on most devices. In addition, touch interaction is mostly limited to single-point operation, which is prone to touch errors. This limits the depth of use in actual interactive scenarios and leads to a poor interactive experience. Therefore, there is room for improvement. Summary of the Invention

[0004] The present application provides a screen display control method, device, computer equipment and storage medium, which can achieve natural interaction while ensuring compatibility and low latency, and effectively improve the user operation experience in the wireless screen projection scenario.

[0005] The above-mentioned invention objective of this application is achieved through the following technical solutions: A screen display control method is applied to a display device, wherein the display device is controlled and connected to a terminal device via wireless technology, and the screen display control method includes: Establishing a wireless connection between the display device and the terminal device, determining a protocol channel between the display device and the terminal device, and receiving screen display data corresponding to the terminal device with which the connection is established; caching the screen display data in a cache area, performing real-time content display according to the screen display data, and determining screen conversion parameters, wherein the screen conversion parameters include a scaling ratio and a position offset; Acquiring user touch action data on the currently displayed content, generating a touch control instruction based on the screen conversion parameter and the touch action data, and sending the touch control instruction to the terminal device through the protocol channel; Receive updated screen display data fed back by the terminal device based on the touch control instruction, and perform real-time content update display according to the updated screen display data.

[0006] By adopting the above technical solution, by establishing a wireless connection between the display device and the terminal device, determining a protocol channel, and receiving screen display data, a stable and reliable data transmission link can be established between the display device and the terminal device, and the initial screen content for screen projection can be obtained. By caching the screen display data in a cache area, performing real-time content display, and determining screen conversion parameters including scaling and position offset, the terminal device screen can be ensured to be presented smoothly and adaptively on the display device, thereby optimizing the user's visual viewing experience. By obtaining the user's touch action data on the currently displayed content, generating touch control instructions based on the screen conversion parameters and touch data and sending them to the terminal device via the protocol channel, the user's intuitive operation on the display device screen can be accurately converted into control intentions for the terminal device and effectively transmitted, thereby realizing direct and convenient reverse control capability of the large screen from the source device. Furthermore, by receiving updated screen display data fed back by the terminal device based on the touch control instructions and performing real-time content update display, the display device screen can be synchronized with the state changes of the terminal device caused by the user operation, thereby providing the user with coherent, low-latency interactive visual feedback, significantly improving the overall smoothness and immersion of the operation.

[0007] In a preferred example, the present application may be further configured as follows: determining the protocol channel between the display device and the terminal device specifically includes: After establishing the wireless connection, performing negotiation interaction of device capabilities and protocols between the display device and the terminal device, and determining support of the standard user input reverse channel by the display device and the terminal device; If both the display device and the terminal device support the standard user input reverse channel, the standard user input reverse channel is used as the protocol channel; If the display device and / or the terminal device does not support the standard user input back channel, a custom data channel is constructed.

[0008] By adopting the above technical solution, by negotiating and interacting the device capabilities and protocols after establishing a wireless connection and judging the support for the standard user input reverse channel, it is possible to dynamically evaluate and identify the most suitable touch command return communication method between the display device and the terminal device. By using the standard channel when both the display device and the terminal device support the standard user input reverse channel, it is possible to give priority to using the communication specifications widely recognized by the industry for data transmission, thereby enhancing the compatibility and interoperability between different devices. By constructing a custom data channel when one or more of the display device or the terminal device does not support the standard user input reverse channel, it is possible to ensure that even in scenarios where some devices cannot support the standardized reverse control protocol, a stable and effective transmission path can still be established, thereby expanding the scope of application.

[0009] In a preferred example, the present application may be further configured as follows: caching the screen display data in a cache area, performing real-time content display according to the screen display data, and determining screen conversion parameters, specifically including: Buffering the screen display data in the buffer area according to a received sequence, processing the screen display data using a preset processing algorithm, and displaying the processed screen display data on the screen in real time; During the real-time content display process, determining a scaling ratio of the displayed content according to a proportional relationship between an original resolution of the terminal device and a display area resolution of the display device; Calculate the position offset of the zoomed display area relative to the center of the screen.

[0010] By adopting the above technical solution, by caching the screen display data in the order of reception and displaying it on the screen in real time after processing it through a preset processing algorithm, the orderly, stable and smooth playback of the terminal projected image can be guaranteed, thereby effectively avoiding problems such as stuttering, misordering or tearing in the image display, and improving the visual quality of the wireless display. By determining the scaling ratio of the display content according to the ratio between the original resolution of the terminal device and the display area resolution of the display device during the real-time content display process, adaptive and precise adjustment of the terminal screen to the display device screen size can be achieved, thereby ensuring that the projected content is presented at the optimal visual size, avoiding information deformation, loss or degradation of the viewing experience caused by improper display ratio. By calculating the position offset of the scaled display area relative to the center of the screen, the optimal layout position of the scaled and adjusted image on the display device screen can be accurately controlled, so that the projected content can be displayed beautifully, centered or aligned in a preset manner, further optimizing the user's overall visual experience.

[0011] In a preferred example, the present application may be further configured as follows: the position offset includes a horizontal offset and a vertical offset, and the calculation of the position offset of the zoomed display area relative to the center of the screen specifically includes: By obtaining the original width and original height of the original screen image of the terminal device, based on the scaling ratio, calculating the actual rendering width and actual rendering height of the screen image after scaling on the display device; By obtaining the display width and display height of the physical display area of ​​the screen of the display device, the horizontal offset and the vertical offset of the scaled display area relative to the center of the screen are calculated based on the position offset calculation formula.

[0012] By adopting the above technical solution, by obtaining the original width and original height of the original screen image of the terminal device and calculating the actual rendering width and actual rendering height of the screen image after scaling on the display device based on the scaling ratio, the specific pixel dimension information that the projected content should occupy on the display target after size adjustment can be accurately obtained. By obtaining the display width and display height of the physical display area of ​​the screen of the display device and calculating the horizontal offset and vertical offset of the scaled display area relative to the center of the screen based on the position offset calculation formula, the specific adjustment values ​​in the precise horizontal and vertical directions required for the scaled image to achieve preset alignment on the screen of the display device can be accurately calculated, thereby ensuring that the projected image can be finely positioned and displayed in strict accordance with the design, thereby improving the regularity and professionalism of the visual layout.

[0013] In a preferred example, the present application may be further configured as follows: acquiring the user's touch action data on the currently displayed content, and generating a touch control instruction based on the screen conversion parameter and the touch action, specifically including: Acquiring a physical touch action performed by a user on the screen, and obtaining touch action data of the user on the currently displayed content based on the physical touch action, the touch action data including touch point coordinates, touch event type, touch state, timestamp, and touch point identifier; Based on the screen conversion parameters, coordinate mapping processing is performed on the touch point coordinates to calculate the target coordinate point of the touch point coordinates in the original screen coordinate system of the terminal device; The touch event type, touch state, timestamp, and touch point identifier corresponding to the target coordinate point and the physical touch action are used to encapsulate the touch control instruction according to a preset rule.

[0014] By adopting the above technical solution, by acquiring the user's physical touch actions on the screen and obtaining touch action data including touch point coordinates, touch event type, touch state, timestamp, and touch point identifier, all original user operation details and interaction intentions on the display device screen can be comprehensively, meticulously, and in real time captured. The touch point coordinates are then mapped based on the screen conversion parameters and the target coordinates in the terminal device's original screen coordinate system are calculated. This allows for accurate and reliable conversion of the physical touch position on the display device to the corresponding position on the terminal device's logical screen, ensuring the accuracy of operation pointing when performing reverse control between devices with different resolutions and display ratios, and avoiding operational errors caused by coordinate mismatches. By encapsulating the target coordinates and the corresponding touch event type, state, timestamp, and touch point identifier into touch control instructions according to preset rules, the calibrated and parsed user operation intentions can be converted into standardized control commands that comply with the communication protocol between the two parties and can be correctly recognized and executed by the terminal device. This ensures the effective generation and reliable transmission of reverse control instructions, completing the closed loop from the user's physical touch on the display device to the corresponding logical function response of the terminal device.

[0015] In a preferred example, the present application may be further configured as follows: receiving the updated screen display data fed back by the terminal device based on the touch control instruction, and performing real-time content update and display according to the updated screen display data, specifically including: receiving, through the protocol channel, updated screen display data fed back by the terminal device based on the touch control instruction, and buffering the data in the buffer area in chronological order; The updated screen display data is processed by the processing algorithm, and the processed screen display data is displayed on the screen in real time.

[0016] By adopting the above technical solution, the updated screen display data based on the touch control instruction feedback of the terminal device is received through the protocol channel and cached in the cache area in chronological order, which can ensure that the display device can obtain and store the latest screen status information generated on the terminal device due to user operations in a timely and orderly manner, and then process the updated screen display data through the processing algorithm and display the processed screen display data on the screen in real time, which can quickly and accurately refresh and present the changes in the screen content of the terminal device on the display device, thereby providing users with low-latency, high-fidelity and continuous visual feedback, significantly enhancing the instant response experience of the entire interactive process and the immersive feeling of user operation.

[0017] In a preferred example, the present application may be further configured as follows: the screen display control method further includes: When at least two touch points are detected, dynamic trajectory recognition is performed on the touch points based on preset gesture recognition rules to identify gesture events of the touch points; encapsulating the gesture event into a gesture control instruction, and sending the gesture control instruction to the terminal device through the protocol channel; After receiving the gesture control instruction, the terminal device performs corresponding interface operation according to the gesture type.

[0018] By adopting the above technical solution, by dynamically identifying the touch points and identifying gesture events based on preset gesture recognition rules when at least two touch points are detected, the display device can accurately parse the user's complex operation intentions such as pinching and sliding completed by a combination of multiple fingers, thereby significantly expanding the interactive dimension of reverse control. By encapsulating gesture events into gesture control instructions and sending them to the terminal device through a protocol channel, the gestures recognized by the display device can be converted into standardized commands that can be understood and executed by the terminal device, and reliably transmitted, thereby ensuring that complex interactive intentions can be conveyed to the terminal device without ambiguity, providing a clear and effective instruction basis for the terminal device to implement corresponding gesture control functions. By enabling the terminal device to perform corresponding interface operations according to the gesture type after receiving the gesture control instruction, the user can conveniently control the terminal device through multi-point gestures directly on the display device, thereby greatly improving the flexibility of operation and providing a better user experience.

[0019] The second object of the present invention is achieved through the following technical solutions: A screen display control device, comprising: a connection and receiving module, configured to establish a wireless connection between the display device and the terminal device, determine a protocol channel between the display device and the terminal device, and receive screen display data corresponding to the terminal device with which the connection is established; A display parameter module is used to cache the screen display data in a cache area, perform real-time content display according to the screen display data, and determine screen conversion parameters, wherein the screen conversion parameters include a scaling ratio and a position offset; a touch control instruction module, configured to obtain user touch action data on the currently displayed content, generate a touch control instruction based on the screen conversion parameter and the touch action data, and send the touch control instruction to the terminal device through the protocol channel; The display update module is used to receive the updated screen display data fed back by the terminal device based on the touch control instruction, and perform real-time content update display according to the updated screen display data.

[0020] By adopting the above technical solution, by establishing a wireless connection between the display device and the terminal device, determining a protocol channel, and receiving screen display data, a stable and reliable data transmission link can be established between the display device and the terminal device, and the initial screen content for screen projection can be obtained. By caching the screen display data in a cache area, performing real-time content display, and determining screen conversion parameters including scaling and position offset, the terminal device screen can be ensured to be presented smoothly and adaptively on the display device, thereby optimizing the user's visual viewing experience. By obtaining the user's touch action data on the currently displayed content, generating touch control instructions based on the screen conversion parameters and touch data and sending them to the terminal device via the protocol channel, the user's intuitive operation on the display device screen can be accurately converted into control intentions for the terminal device and effectively transmitted, thereby realizing direct and convenient reverse control capability of the large screen from the source device. Furthermore, by receiving updated screen display data fed back by the terminal device based on the touch control instructions and performing real-time content update display, the display device screen can be synchronized with the state changes of the terminal device caused by the user operation, thereby providing the user with coherent, low-latency interactive visual feedback, significantly improving the overall smoothness and immersion of the operation.

[0021] The third objective of this application is achieved through the following technical solutions: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned screen display control method are implemented.

[0022] The fourth objective of this application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned screen display control method when executed by a processor.

[0023] In summary, this application includes at least one of the following beneficial technical effects: 1. By establishing a wireless connection between the display device and the terminal device, determining a protocol channel, and receiving screen display data, a stable and reliable data transmission link can be established between the display device and the terminal device, and the initial screen content for screen projection can be obtained. By caching the screen display data in a cache area, real-time content display and determining screen conversion parameters including scaling and position offset, the terminal device screen can be smoothly and adaptively presented on the display device, thereby optimizing the user's visual viewing experience. By obtaining the user's touch action data on the currently displayed content, generating touch control instructions based on the screen conversion parameters and touch data and sending them to the terminal device through the protocol channel, the user's intuitive operation on the display device screen can be accurately converted into control intentions for the terminal device and effectively transmitted, thereby realizing direct and convenient reverse control capability of the large screen from the source device. Furthermore, by receiving updated screen display data fed back by the terminal device based on the touch control instructions and performing real-time content update display, the display device screen can be synchronized with the terminal device state changes caused by user operation, thereby providing users with consistent, low-latency interactive visual feedback, significantly improving the overall smoothness and immersion of operation. 2. By dynamically identifying the touch points and gesture events based on preset gesture recognition rules when at least two touch points are detected, the display device can accurately interpret the user's complex operation intentions such as pinching and sliding performed by a combination of multiple fingers, thereby significantly expanding the interactive dimension of reverse control. By encapsulating gesture events into gesture control instructions and sending them to the terminal device through a protocol channel, the gestures recognized by the display device can be converted into standardized commands that can be understood and executed by the terminal device and transmitted reliably, thereby ensuring that complex interaction intentions can be conveyed to the terminal device without ambiguity, providing a clear and effective instruction basis for the terminal device to implement corresponding gesture control functions. By enabling the terminal device to perform corresponding interface operations according to the gesture type after receiving the gesture control instruction, the user can conveniently control the terminal device through multi-point gestures directly on the display device, thereby greatly improving the flexibility of operation and providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of a screen display control method according to an embodiment of the present application; Figure 2 This is a flowchart for implementing step S10 in a screen display control method in one embodiment of the present application; Figure 3 This is a flowchart for implementing step S10 in a screen display control method in one embodiment of the present application; Figure 4 This is a flowchart for implementing step S10 in a screen display control method in one embodiment of the present application; Figure 5 This is a flowchart for implementing step S10 in a screen display control method in one embodiment of the present application; Figure 6 This is a flowchart for implementing step S10 in a screen display control method in one embodiment of the present application; Figure 7 This is another implementation flow chart of a screen display control method in one embodiment of the present application; Figure 8 This is a principle block diagram of a screen display control device in one embodiment of the present application; Figure 9 It is a schematic diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following examples will help those skilled in the art further understand the purpose of this application, but are not intended to limit this application in any form. It should be noted that those skilled in the art may make several modifications and improvements without departing from the scope of this application. These modifications and improvements are all within the scope of this application.

[0026] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0028] The present application is further described in detail below with reference to the accompanying drawings.

[0029] In one embodiment, if Figure 1 As shown, the present application discloses a screen display control method, which specifically includes the following steps: S10: establishing a wireless connection between the display device and the terminal device, determining a protocol channel between the display device and the terminal device, and receiving screen display data corresponding to the terminal device with which the connection is established.

[0030] Specifically, a direct point-to-point connection is first established between the display device and the terminal device through the built-in WiFi module, where the display device can be a touch display and the terminal device can be a smartphone. The point-to-point connection method can be achieved by starting the WiFi Direct function to discover and pair devices, or by performing service discovery in a unified local area network through a local broadcast protocol such as mDNS. After the connection is established, the display device and the terminal device will conduct preliminary communication negotiation to determine the basic rules and channel characteristics for data transmission. The negotiation process aims to clarify the logical paths used for subsequent screen image data streams and user input control instructions respectively. When the negotiation is completed and the protocol channel is confirmed, the display device begins to receive the screen image data stream currently output by the terminal device from the terminal device through the protocol channel. This data is usually a compressed and encoded video frame sequence.

[0031] S20: Cache the screen display data in a cache area, perform real-time content display according to the screen display data, and determine screen conversion parameters, which include a scaling ratio and a position offset.

[0032] Specifically, after receiving the screen display data from the terminal device, the display device will first temporarily store the screen display data, such as H.264-encoded video frames, in the internal memory cache to cope with possible network transmission fluctuations and ensure the smoothness of subsequent decoding and display. At the same time, the data in the cache will be decoded and restored into displayable image frames, and rendered and presented in real time on the display device screen. During the display process, in order to make the picture of the terminal device properly adapt to the screen size and resolution of the display device, necessary picture adjustment calculations will be performed to determine one or more picture conversion parameters for guiding how the picture is scaled and how it is positioned on the screen, such as determining the multiple by which the original picture needs to be enlarged or reduced and the key parameters of the adjusted picture on the screen.

[0033] S30: Acquire the user's touch action data on the currently displayed content, generate a touch control instruction based on the screen conversion parameter and the touch action data, and send the touch control instruction to the terminal device through the protocol channel.

[0034] Specifically, when the user taps, slides, or performs other operations on the touch screen of the display device, the touch sensing module of the display device will instantly capture these physical actions and convert them into touch action data containing information such as touch position and time. Then, using the determined image conversion parameters, such as the calculated scaling factor and the offset value of the image on the screen, the original touch point coordinates are subjected to necessary mathematical transformations to calculate the corresponding position of the touch point in the original screen coordinate system of the terminal device. The mapped target coordinates and related touch event information are encapsulated into one or more control instructions in accordance with the format required by the determined protocol channel, and wirelessly sent to the terminal device through the protocol channel to instruct it to perform the corresponding operation.

[0035] S40: receiving updated screen display data fed back by the terminal device based on the touch control instruction, and performing real-time content update and display according to the updated screen display data.

[0036] Specifically, after the terminal device executes the received touch control instructions, the screen display content will usually change, such as an application being opened or a list being scrolled. At the same time, the terminal device will immediately generate these changes as new screen display data and send it back to the display device through the protocol channel. After the display device receives these updated data, it will quickly replace or supplement the old content in the cache area and synchronously refresh the display on the screen, allowing the user to see the real-time feedback effect brought by the touch operation.

[0037] In one embodiment, if Figure 2 As shown, in step S10, the protocol channel between the display device and the terminal device is determined, which specifically includes: S11: After the wireless connection is established, negotiation and interaction of device capabilities and protocols are performed between the display device and the terminal device to determine whether the display device and the terminal device support the standard user input reverse channel.

[0038] Specifically, after the wireless connection is successfully established, the display device and the terminal device will enter a device capability and protocol negotiation phase. For example, parameters can be exchanged based on the RTSP (Real-Time Streaming Protocol) session in the Miracast standard. Both parties will inform each other of the features they support, such as video codec capabilities, screen resolution, and, most importantly, whether the standard User Input Back Channel (UIBC) function is supported or opened. The display device will explicitly query the terminal device's support for standard reverse input protocols such as Miracast UIBC general mode or HIDC (HID Command) mode.

[0039] S12: If both the display device and the terminal device support the standard user input reverse channel, the standard user input reverse channel is used as the protocol channel.

[0040] Specifically, in the previous negotiation interaction process, the analysis results show that both the display device and the terminal device declare support and agree to enable the same standard user input reverse channel. For example, if both parties confirm support for Miracast UIBC and the terminal device has enabled this function, then the standard channel, such as the UIBC channel, will be selected as the protocol channel for subsequently transmitting touch control instructions from the display device to the terminal device, so that the existing standardized framework can be used for data transmission.

[0041] S13: If the display device and / or the terminal device does not support a standard user input back channel, a custom data channel is constructed.

[0042] Specifically, if during the negotiation process it is discovered that the terminal device explicitly indicates that it does not support UIBC, or its UIBC function is not open or has compatibility issues, or the display device itself is not inclined to use UIBC, the two parties will negotiate to establish a custom second data channel. For example, the display device can request that the terminal device open a specific TCP port listening connection at the operating system level, or the two parties can agree to use UDP for datagram transmission. The display device then establishes an independent TCP connection or UDP communication path with the terminal device through this negotiated port, specifically for transmitting encapsulated touch command data packets. The data format and communication logic of this custom channel are agreed upon by both parties.

[0043] In one embodiment, if Figure 3 As shown, in step S20, the screen display data is cached in the cache area, real-time content is displayed according to the screen display data, and screen conversion parameters are determined, specifically including: S21: Cache the screen display data in a cache area according to the received sequence, process the screen display data using a preset processing algorithm, and display the processed screen display data on the screen in real time.

[0044] Specifically, each frame of screen display data received from the terminal device will be assigned a serial number or timestamp, and stored in a pre-allocated ring buffer (RingBuffer) or first-in-first-out (FIFO) queue in the display device in strict accordance with the order of these identifiers to ensure the timing correctness of the picture. Data frames are continuously taken out from the head of the buffer area and processed through built-in image processing, which includes at least a decoding module such as an H.264 decoder, a color space conversion module such as YUV to RGB, and an image enhancement module such as sharpening or contrast adjustment. Finally, the rendered image data is sent to the display panel for real-time screen display.

[0045] The terminal device will capture its screen image, for example, at a rate of 30 frames per second (30fps), compress it into a video frame sequence in real time through a hardware-accelerated video encoder (such as H.264 or H.265), and send it to the display device through a protocol channel (for example, using RTP / UDP or TCP protocol). After receiving these encoded video data streams, the display device will store them in an internal video frame buffer, i.e., a cache area, in the order received, usually based on the frame number or timestamp. Then, through built-in image processing, which includes at least a decoding module such as an H.264 decoder, a color space conversion module such as YUV to RGB, and an image enhancement module such as sharpening or contrast adjustment, the video frames in the buffer area will be decoded and the decoded original image data will be displayed on the screen, thereby realizing mirroring of the terminal device screen.

[0046] S22: During the real-time content display process, a scaling ratio of the displayed content is determined according to a proportional relationship between the original resolution of the terminal device and the display area resolution of the display device.

[0047] Specifically, when displaying a mirrored terminal device screen, in order to adapt to the physical size and resolution of the display screen, for example, the original resolution of the terminal device may be 1080x2340, while the display resolution may be 1920x1080 or have a specific display window area, the original width and height of the terminal device screen will be obtained and compared with the width and height of the currently available display area of ​​the display. Based on the principle of maintaining the original aspect ratio of the terminal device screen, a unified scaling factor is calculated. For example, if the terminal device screen is to be fully displayed, the ratio of the available width of the display to the original width of the terminal device and the ratio of the available height of the display to the original height of the terminal device may be taken. The smaller of these two ratios is used as the final scaling ratio to ensure that the screen content is not cropped.

[0048] S23: Calculate the position offset of the zoomed display area relative to the center of the screen.

[0049] Specifically, after the terminal device screen is scaled according to a determined scaling ratio, the size of the screen content finally presented on the display may not be able to completely fill the physical display area of ​​the display. For example, when the aspect ratio of the terminal device screen is different from the aspect ratio of the display, the boundary will be touched in one dimension after scaling while there is still space in the other dimension. At this time, in order to make the scaled terminal device screen displayed in the center of the display screen or displayed according to a preset alignment method, such as top alignment and left alignment, it is necessary to calculate the horizontal and vertical offsets of the upper left corner vertex of the display area of ​​the scaled screen relative to the upper left corner vertex or center point of the physical display area of ​​the display. These offsets will be used to accurately position the scaled screen during subsequent rendering.

[0050] In one embodiment, if Figure 4 As shown, in step S23, the position offset includes a horizontal offset and a vertical offset, and the position offset of the zoomed display area relative to the center of the screen is calculated, specifically including: S231: By obtaining the original width and original height of the original screen image of the terminal device, based on the scaling ratio, the actual rendering width and actual rendering height of the screen image after scaling on the display device are calculated.

[0051] Specifically, first obtain the original size of the output image from the terminal device or record it when the connection is established, for example, the original width is W orig For example, 1080 pixels, the original height is H orig For example, 1920 pixels, and then use the uniform scaling ratio S determined earlier, such as 0.5, to calculate the actual width W of the image to be rendered on the display device. scaled =W orig ×S, such as W scaled is 540 pixels, and the actual height H scaled =H orig ×S, such as H scaled For 960 pixels, these calculations represent the pixel dimensions that the content will occupy on the display after scaling.

[0052] S232: Obtain the display width and display height of the physical display area of ​​the display device, and calculate the horizontal offset and vertical offset of the scaled display area relative to the center of the screen based on a position offset calculation formula.

[0053] Specifically, query the physical display parameters of the display device itself to obtain the total width W of the actual screen display area of ​​the display device. disp Such as 1280 pixels, and the total height H disp For example, 720 pixels, by applying the center alignment position offset calculation formula, such as the horizontal offset Offset X =(Wdisp −W scaled ) / 2, such as Offset X The vertical offset is (1280−540) / 2=370 pixels. Y =(H disp −H scaled ) / 2, such as Offset Y = (720−960) / 2=−120 pixels. Here, you may need to adjust the strategy to avoid negative offsets, or to indicate that the part of the content that exceeds the limit is cropped or scrolled from a specific edge. If the upper left corner is (0,0), the offset is usually positive to ensure that the content is within the display area. These calculated offsets (Offset X , Offset Y ) together with the scaling ratio S are recorded as the screen conversion parameters for subsequent touch coordinate mapping.

[0054] In one embodiment, if Figure 5 As shown, in step S30, the user's touch action data on the currently displayed content is obtained, and a touch control instruction is generated based on the screen conversion parameter and the touch action, which specifically includes: S31: Acquire a physical touch action performed by a user on the screen, and obtain touch action data of the user on the currently displayed content based on the physical touch action. The touch action data includes touch point coordinates, touch event type, touch state, timestamp, and touch point identifier.

[0055] Specifically, the built-in touch acquisition module of the display device will continuously monitor whether there is any touch on its screen surface. When the user's finger or other touch tool contacts, moves or leaves the screen surface, the module will immediately capture these physical touch events. For example, when the user's finger presses the screen, the detailed parameters of the touch event will be captured immediately. For each event, a set of detailed touch action data will be generated, where the touch action data includes the X and Y coordinate values ​​of the touch point in the display device screen coordinate system, indicating the position of the touch point, such as (100, 200), and records the type of the event, such as whether it is pressed, slid or lifted, and the current state of the touch, such as whether it is the main touch point, and a timestamp accurate to milliseconds is given to the event, recording the time when the event occurred, as well as a unique touch identification ID, which is used to track and distinguish different touch points in multi-touch scenarios.

[0056] S32: Based on the screen conversion parameters, coordinate mapping processing is performed on the touch point coordinates to calculate the target coordinate point of the touch point coordinates in the original screen coordinate system of the terminal device.

[0057] Specifically, by using the determined picture conversion parameters, namely the scaling ratio S and the position offset (OffsetX ,Offset Y ), based on the acquired display device touch point coordinates (X touch ,Y touch ) to perform reverse conversion operations, for example, the X coordinate of the target touch point in the original screen coordinate system of the terminal device can be obtained by X target =(X touch −Offset X ) / S, similarly, the target Y coordinate can be calculated by Y target =(Y touch −Offset Y ) / S is calculated, so the calculated (X targeth ,Y target ) is the coordinate point corresponding to the original screen resolution of the terminal device. This coordinate mapping process ensures that even if the user touches the scaled and offset display content, the touch can accurately act on the corresponding position of the original terminal screen, avoiding inaccurate operation.

[0058] S33: Encapsulate the touch event type, touch state, timestamp, and touch point identifier corresponding to the target coordinate point and the physical touch action into a touch control instruction according to a preset rule.

[0059] Specifically, the target coordinate point (X targeth ,Y target ), and the event type corresponding to the acquired physical touch action, such as press, move, lift, touch status, timestamp, and contact identification ID, are integrated together, and organized and serialized according to the data structure specified by the protocol channel determined by both parties. For example, if it is the Generic mode or HIDC mode of Miracast UIBC, it is encapsulated according to its specified data packet structure, or if it is a custom TCP channel, it is encapsulated according to the custom data format agreed upon by both parties, such as a JSON string or a specific binary structure, to form a control signal data packet containing complete touch information, that is, a touch control instruction.

[0060] In one embodiment, if Figure 6 As shown, in step S40, that is, receiving the updated screen display data fed back by the terminal device based on the touch control instruction, and performing real-time content update display according to the updated screen display data, specifically including: S41: receiving updated screen display data fed back by the terminal device based on the touch control instruction through the protocol channel, and buffering the data into the buffer area in chronological order.

[0061] Specifically, the display device will continuously receive screen data updates from the terminal device through the established protocol channel. When the terminal device changes the display content due to executing the touch control instruction, it will send the new screen image frame or difference data through the protocol channel. These updated data are usually new video frames containing the content of the picture changes. Each frame of data has a serial number or timestamp information. When the display device receives it, it will be stored in the corresponding position of the internal screen display data cache area in an orderly manner according to its chronological order, usually overwriting the old same-sequence frame or inserting it to the end of the playback queue to ensure the continuity and consistency of the picture during subsequent display.

[0062] S42: Processing the updated screen display data through a processing algorithm, and displaying the processed screen display data on the screen in real time.

[0063] Specifically, the latest screen display data received and cached in the correct order is obtained from the cache area. These updated data will also undergo a preset image processing algorithm, including video decoding such as H.264 / H.265 decoding, color space conversion and other steps to restore the encoded video data into image data that can be directly displayed. After that, these processed image data are immediately sent to the display panel of the display device for refresh, so that the latest changes in the terminal device interface caused by the user's touch operation are presented on the screen in real time, ensuring the continuity and immediacy of the interaction.

[0064] In one embodiment, if Figure 7 As shown, after step S31, the screen display control method further includes: S301: When at least two touch points are detected, dynamic trajectory recognition is performed on the touch points based on preset gesture recognition rules to identify gesture events of the touch points.

[0065] Specifically, after obtaining the user's physical touch action data, the number of currently active touch points will be checked. If at least two different contact identification IDs are detected on the screen at the same time, multi-touch gesture recognition will be started. The multi-touch gesture recognition will track in real time the position coordinate sequence, movement speed, and relative distance changes of these two or more touch points on the display device screen, such as whether the distance between the two points increases or decreases, and their respective touch states, such as the timing relationship of pressing, moving, and lifting. Then, these dynamic trajectory information will be matched with a series of preset gesture recognition rules, for example, pinching two fingers together to zoom out, spreading two fingers apart to zoom in, or quickly sliding two fingers in the same direction to turn pages, etc. These dynamic trajectory data will be pattern matched. Once the match is successful, a specific gesture event, such as a pinch-to-zoom gesture, will be recognized.

[0066] S302: Encapsulate the gesture event into a gesture control instruction, and send the gesture control instruction to the terminal device through a protocol channel.

[0067] Specifically, once a gesture event is successfully recognized, such as a two-finger pinch gesture, and the zoom center point and zoom scale factor are calculated, the center point coordinates also need to be converted to the terminal device coordinate system through coordinate mapping, and the gesture event type, such as pinch_zoom, and related parameters, such as the center point coordinates (X center_target ,Y center_target) , scaling factor F actorscale Etc., encapsulate the data format according to the protocol format requirements of the determined protocol channel, such as standard UIBC or custom channel, encapsulate it into a specific gesture control instruction, and then send it to the terminal device through the protocol channel.

[0068] S303: After receiving the gesture control instruction, the terminal device performs corresponding interface operation according to the gesture type.

[0069] Specifically, after the terminal device receives the gesture control instruction from the display device, it will parse the gesture control instruction and extract the gesture types contained therein, such as two-finger pinching, two-finger dragging, etc., as well as related parameters such as center point, zoom ratio, displacement vector, etc. The terminal device will call the corresponding input interface provided by the terminal device operating system level, such as the Input API or Accessibility Service provided by the general operating system, or simulate HID device input to simulate an equivalent local gesture operation. After the terminal device operating system or the currently active application receives this simulated input event, it will execute the interface operation corresponding to the gesture type, such as zooming in and out of an image, dragging on a map, or scrolling in a list, etc., thereby responding to the user's multi-touch gesture.

[0070] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0071] In one embodiment, a screen display control device is provided, which corresponds one-to-one to the screen display control method in the above embodiment. Figure 8 As shown, the screen display control device includes a connection and receiving module, a display parameter module, a touch instruction module and a display update module. The detailed description of each functional module is as follows: The connection and receiving module is used to establish a wireless connection between the display device and the terminal device, determine the protocol channel between the display device and the terminal device, and receive the screen display data corresponding to the terminal device with which the connection is established; A display parameter module is used to cache screen display data in a cache area, perform real-time content display based on the screen display data, and determine screen conversion parameters, which include scaling and position offset; The touch command module is used to obtain the user's touch action data on the currently displayed content, generate touch control instructions based on the screen conversion parameters and the touch action data, and send the touch control instructions to the terminal device through the protocol channel; The display update module is used to receive the updated screen display data fed back by the terminal device based on the touch control instruction, and perform real-time content update and display according to the updated screen display data.

[0072] Optionally, the connection and receiving module specifically includes: The channel negotiation submodule is used to negotiate the device capabilities and protocols between the display device and the terminal device after the wireless connection is established, and to determine whether the display device and the terminal device support the standard user input reverse channel; The standard channel submodule is used to use the standard user input reverse channel as the protocol channel if both the display device and the terminal device support the standard user input reverse channel; The custom channel submodule is used to build a custom data channel if the display device and / or terminal device does not support the standard user input back channel.

[0073] Optionally, the display parameter module specifically includes: The data display submodule is used to cache the screen display data in a cache area according to the received sequence, process the screen display data through a preset processing algorithm, and display the processed screen display data on the screen in real time; The scaling submodule is used to determine the scaling ratio of the displayed content according to the proportional relationship between the original resolution of the terminal device and the display area resolution of the display device during the real-time content display process; The position offset submodule is used to calculate the position offset of the zoomed display area relative to the center of the screen.

[0074] Optionally, the position offset submodule specifically includes: The rendering size unit is used to obtain the original width and original height of the original screen image of the terminal device and calculate the actual rendering width and actual rendering height of the screen image after scaling on the display device based on the scaling ratio; The offset calculation unit is used to obtain the display width and display height of the physical display area of ​​the screen of the display device, and calculate the horizontal offset and vertical offset of the scaled display area relative to the center of the screen based on the position offset calculation formula.

[0075] Optionally, the touch command module specifically includes: The touch acquisition submodule is used to obtain the user's physical touch actions on the screen and obtain the user's touch action data on the currently displayed content based on the physical touch actions. The touch action data includes the touch point coordinates, touch event type, touch state, timestamp, and touch point identifier; The coordinate mapping submodule is used to perform coordinate mapping processing on the touch point coordinates based on the screen conversion parameters, and calculate the target coordinate point of the touch point coordinates in the original screen coordinate system of the terminal device; The instruction encapsulation submodule is used to encapsulate the touch event type, touch state, timestamp and touch point identifier corresponding to the target coordinate point and the physical touch action into a touch control instruction according to preset rules.

[0076] Optionally, the display update module specifically includes: The update receiving submodule is used to receive the updated screen display data fed back by the terminal device based on the touch control command through the protocol channel, and cache it in the cache area in chronological order; The update display submodule is used to process the update screen display data through a processing algorithm and display the processed screen display data on the screen in real time.

[0077] Optionally, the screen display control device further includes: A gesture recognition module is used to perform dynamic trajectory recognition on the touch points and identify gesture events of the touch points based on preset gesture recognition rules when at least two touch points are detected; The gesture command module is used to encapsulate gesture events into gesture control commands and send the gesture control commands to the terminal device through the protocol channel; The gesture response module is used to perform corresponding interface operations according to the gesture type after the terminal device receives the gesture control command.

[0078] For the specific definition of the screen display control device, please refer to the definition of the screen display control method above, which will not be repeated here. The various modules in the above-mentioned screen display control device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0079] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data information such as screen display data, screen conversion parameters and touch control instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a screen display control method is implemented.

[0080] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Establishing a wireless connection between the display device and the terminal device, determining a protocol channel between the display device and the terminal device, and receiving screen display data corresponding to the terminal device with which the connection is established; Cache the screen display data in a cache area, display the content in real time according to the screen display data, and determine the screen conversion parameters, which include a scaling ratio and a position offset; Acquire the user's touch action data on the currently displayed content, generate touch control instructions based on the screen conversion parameters and the touch action data, and send the touch control instructions to the terminal device through the protocol channel; The receiving terminal device updates the screen display data based on the touch control instruction feedback, and performs real-time content update and display according to the updated screen display data.

[0081] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Establishing a wireless connection between the display device and the terminal device, determining a protocol channel between the display device and the terminal device, and receiving screen display data corresponding to the terminal device with which the connection is established; Cache the screen display data in a cache area, display the content in real time according to the screen display data, and determine the screen conversion parameters, which include a scaling ratio and a position offset; Acquire the user's touch action data on the currently displayed content, generate touch control instructions based on the screen conversion parameters and the touch action data, and send the touch control instructions to the terminal device through the protocol channel; The receiving terminal device updates the screen display data based on the touch control instruction feedback, and performs real-time content update and display according to the updated screen display data.

[0082] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0084] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A screen display control method, characterized in that: Applied to a display device, the display device is connected to a terminal device through wireless technology, and the screen display control method includes: Establishing a wireless connection between the display device and the terminal device, determining a protocol channel between the display device and the terminal device, and receiving screen display data corresponding to the terminal device with which the connection is established; caching the screen display data in a cache area, performing real-time content display according to the screen display data, and determining screen conversion parameters, wherein the screen conversion parameters include a scaling ratio and a position offset; Acquiring user touch action data on the currently displayed content, generating a touch control instruction based on the screen conversion parameter and the touch action data, and sending the touch control instruction to the terminal device through the protocol channel; Receive updated screen display data fed back by the terminal device based on the touch control instruction, and perform real-time content update display according to the updated screen display data.

2. The screen display control method according to claim 1, wherein: The determining of the protocol channel between the display device and the terminal device specifically includes: After establishing the wireless connection, performing negotiation interaction of device capabilities and protocols between the display device and the terminal device, and determining support of the standard user input reverse channel by the display device and the terminal device; If both the display device and the terminal device support the standard user input reverse channel, the standard user input reverse channel is used as the protocol channel; If the display device and / or the terminal device does not support the standard user input back channel, a custom data channel is constructed.

3. The screen display control method according to claim 1, wherein: The step of caching the screen display data in a cache area, performing real-time content display according to the screen display data, and determining screen conversion parameters specifically includes: Buffering the screen display data in the buffer area according to a received sequence, processing the screen display data using a preset processing algorithm, and displaying the processed screen display data on the screen in real time; During the real-time content display process, determining a scaling ratio of the displayed content according to a proportional relationship between an original resolution of the terminal device and a display area resolution of the display device; Calculate the position offset of the zoomed display area relative to the center of the screen.

4. The screen display control method according to claim 3, characterized in that: The position offset includes a horizontal offset and a vertical offset. The calculation of the position offset of the zoomed display area relative to the center of the screen specifically includes: By obtaining the original width and original height of the original screen image of the terminal device, based on the scaling ratio, calculating the actual rendering width and actual rendering height of the screen image after scaling on the display device; By obtaining the display width and display height of the physical display area of ​​the screen of the display device, the horizontal offset and the vertical offset of the scaled display area relative to the center of the screen are calculated based on the position offset calculation formula.

5. The screen display control method according to claim 1, characterized in that: The acquiring of the user's touch action data on the currently displayed content and generating a touch control instruction based on the screen conversion parameter and the touch action specifically includes: Acquiring a physical touch action performed by a user on the screen, and obtaining touch action data of the user on the currently displayed content based on the physical touch action, the touch action data including touch point coordinates, touch event type, touch state, timestamp, and touch point identifier; Based on the screen conversion parameters, coordinate mapping processing is performed on the touch point coordinates to calculate the target coordinate point of the touch point coordinates in the original screen coordinate system of the terminal device; The touch event type, touch state, timestamp, and touch point identifier corresponding to the target coordinate point and the physical touch action are used to encapsulate the touch control instruction according to a preset rule.

6. The screen display control method according to any one of claims 1 to 5, characterized in that: The receiving updated screen display data fed back by the terminal device based on the touch control instruction, and performing real-time content update and display according to the updated screen display data specifically includes: receiving, via the protocol channel, updated screen display data fed back by the terminal device based on the touch control instruction, and buffering the data in the buffer area in chronological order; The updated screen display data is processed by the processing algorithm, and the processed screen display data is displayed on the screen in real time.

7. The screen display control method according to claim 1, characterized in that: The screen display control method further includes: When at least two touch points are detected, dynamic trajectory recognition is performed on the touch points based on preset gesture recognition rules to identify gesture events of the touch points; encapsulating the gesture event into a gesture control instruction, and sending the gesture control instruction to the terminal device through the protocol channel; After receiving the gesture control instruction, the terminal device performs corresponding interface operation according to the gesture type.

8. A screen display control device, characterized in that: The screen display control device comprises: a connection and receiving module, configured to establish a wireless connection between the display device and the terminal device, determine a protocol channel between the display device and the terminal device, and receive screen display data corresponding to the terminal device with which the connection is established; A display parameter module is used to cache the screen display data in a cache area, perform real-time content display according to the screen display data, and determine screen conversion parameters, wherein the screen conversion parameters include a scaling ratio and a position offset; a touch control instruction module, configured to obtain user touch action data on the currently displayed content, generate a touch control instruction based on the screen conversion parameter and the touch action data, and send the touch control instruction to the terminal device through the protocol channel; The display update module is used to receive the updated screen display data fed back by the terminal device based on the touch control instruction, and perform real-time content update display according to the updated screen display data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the screen display control method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the screen display control method according to any one of claims 1 to 7 are implemented.

Images