A method for screen projection and reverse control of a dual-mode browser in a restricted network environment and a mobile terminal sending end

Abstract

This invention discloses a dual-mode browser screen projection and reverse control method in a restricted network environment. The mobile terminal sending end collects screen video data, provides a browser receiving page to the browser receiving terminal, and receives the WebRTC and MediaSource capabilities reported by the browser receiving terminal. The mobile terminal sending end selects either a WebRTC transmission link or an MSE media buffer transmission link based on the capability vector and operating strategy, and binds either RTCDataChannel or WebSocket as the control return channel, both carrying unified control data of the same format. When direct access to private addresses is restricted, the mobile terminal sending end returns the receiving page through hotspot, preset public IP address, or domain name request identification and local routing. The browser receiving page can pre-map pointer events to source screen coordinates, and the mobile terminal sending end then performs rotation transformation and executes reverse control. (See Figure 1 for abstract and accompanying figures.)

Description

Technical Field

[0001] This invention relates to the fields of mobile terminal screen capture, browser media transmission, restricted network access, control feedback, and human-computer interaction coordinate mapping, specifically to a dual-mode browser screen projection and reverse control method and a mobile terminal transmitter in a restricted network environment. Background Technology

[0002] Mobile terminals can capture screen images after obtaining screen capture authorization and generate video data that can be transmitted over the network through an encoder. Browser receiving terminals typically have a browser runtime environment, which can load web page scripts and use interfaces such as WebRTC, RTCDataChannel, MediaSource, SourceBuffer, and WebSocket to implement media reception and interactive control.

[0003] Existing screen mirroring solutions mostly employ a single transmission path. Real-time interactive scenarios tend to use the WebRTC link, while high-compatibility playback scenarios tend to use HTTP media clips and MediaSource append playback links. When browser capabilities, network environment, security context, and codec support are inconsistent, the single-path solution is prone to problems such as connection establishment failure, excessively long first-frame wait, unavailable control channels, or missing fallback paths.

[0004] In some scenarios involving in-vehicle central control screens, embedded browsers, or restricted intranet browsers, the browser receiving terminal may not be able to directly input or access the mobile terminal's private address, or may not be able to open the local screen mirroring service through the conventional LAN discovery mechanism. In this case, even if the mobile terminal sending end has already run the screen mirroring service, the browser receiving terminal may still be unable to reach the receiving page.

[0005] Existing remote control solutions often directly convert browser event coordinates to terminal coordinates according to the display ratio, without fully considering factors such as horizontal or vertical black borders caused by proportional scaling of video elements, inconsistencies between the source screen orientation and the target screen's current rotation direction, and differences in the permissions of the browser receiving page and the mobile terminal's input execution path. This results in click offsets, incorrect swipe directions, or the inability to reuse control data in different media modes.

[0006] Therefore, a browser-based screen mirroring solution that can work in both ordinary LAN and restricted network environments is needed, so that media transmission link selection, restricted network access, control backhaul channel, unified control data and coordinate mapping can form a stable technical closed loop. Summary of the Invention

[0007] The purpose of this invention is to provide a dual-mode browser screen projection and reverse control scheme led by the mobile terminal sending end, in order to solve the technical problems of heterogeneous browser capabilities, difficulty in accessing restricted networks, lack of fallback after failure of a single media link, inconsistent control backhaul protocols, and inaccurate reverse control coordinate mapping.

[0008] To achieve the above objectives, after receiving authorization for screen capture, the mobile terminal sending end establishes a virtual display and captures screen video frames, encoding these frames into video data. The mobile terminal sending end provides a browser receiving page to the browser receiving terminal, and the browser runtime environment in the browser receiving terminal loads this browser receiving page. The browser receiving page performs capability detection and reports a capability vector to the mobile terminal sending end. The capability vector includes a first capability set for determining the availability of the WebRTC transmission link and a second capability set for determining the availability of the MSE media buffer transmission link.

[0009] The mode selection module in the mobile terminal transmitter generates a target transmission link from candidate media transmission links, including WebRTC transmission links and MSE media buffer transmission links, based on the capability vector and operating strategy parameters. It then establishes corresponding media output channels and control return channels based on the target transmission link. The control return channels are bound to either RTCDataChannel or WebSocket, and different control return channels carry unified control data of the same format. The mobile terminal transmitter receives the unified control data through the control return channels and, based on the control information parsed from the unified control data, invokes the input execution path to complete reverse control.

[0010] When the browser receiving terminal's direct access to the private address of the mobile terminal sending terminal is restricted, the restricted network access module in the mobile terminal sending terminal performs the restricted network access steps, which specifically include: turning on a wireless hotspot for the browser receiving terminal to access, identifying the access request initiated by the browser receiving terminal to a preset public address or domain name, and routing the access request to the local receiving page service through local DNS rewriting, transparent proxy, or virtual private network traffic diversion, and the local receiving page service returning the browser receiving page and session parameters.

[0011] Furthermore, the browser receiving page can pre-map the user pointer event position to source screen coordinates based on the video element boundaries, source screen size, and the black border offset caused by proportional display, and encapsulate this into unified control data. After receiving the unified control data, the mobile terminal sending end first normalizes the source screen coordinates according to the source screen size, and then calculates the target horizontal and vertical coordinates according to the coordinate transformation relationships corresponding to 0 degrees, 90 degrees, 180 degrees, or 270 degrees based on the current rotation direction. This yields the target control coordinates, and the input execution path matching the deployment permissions is invoked to complete click, swipe, button, or text control.

[0012] Compared with existing technologies, this invention has at least the following advantages. First, independent capability detection and operation strategy selection enable the WebRTC transmission link and the MSE media buffer transmission link to switch according to the scenario, improving the success rate of establishment in heterogeneous browser environments. Second, while using RTCDataChannel and WebSocket respectively on the control return channel, the semantics and format of the unified control data remain consistent, reducing the control stack complexity during media mode switching. Third, the restricted network access module, through hotspot access, access request identification, and local routing, enables browser receiving terminals that cannot directly access private addresses to also obtain the receiving page. Fourth, browser-side pre-mapping and mobile terminal-side rotation transformation can reduce control offset caused by black borders, scaling, and screen orientation changes. Fifth, the session identifier, event sequence number, and active window resource release mechanism help reduce miscontrol, replay control, and idle resource occupation. Attached Figure Description

[0013] Figure 1 This is the overall system architecture diagram.

[0014] Figure 2 This is a flowchart for dual-mode selection and rollback.

[0015] Figure 3 This is a schematic diagram of the WebRTC transmission link interaction.

[0016] Figure 4 Flowchart of the MSE media buffer transmission link.

[0017] Figure 5 A flowchart for the browser to receive pre-mapped content from a page.

[0018] Figure 6 This is a flowchart of the normalization and rotation transformation in the mobile terminal's transmitting end.

[0019] Figure 7 Flowchart for selecting the reverse control execution path.

[0020] Figure 8 This is a flowchart for receiving entry point discovery, pairing, and resource release.

[0021] Figure 9 Flowchart for redirecting access to restricted network hotspots.

[0022] Figure 10 This is a diagram showing the component relationships of a restricted network access module.

[0023] Figure labeling: 100 Mobile terminal transmitter; 110 Screen / audio acquisition module; 120 Coordinate mapping execution module; 130 Encoding / encapsulation service module; 140 Service discovery and pairing module; 150 Restricted network access module; 151 Hotspot management unit; 152 Access request identification unit; 153 Local routing unit; 154 Local receiving page service unit; 155 Loop protection unit; 200 Browser receiving terminal; 201 Browser operating environment; 202 Browser receiving page; 210 Capability detection module; 220 Playback rendering module; 230 Browser pre-mapping module; 240 Unified control data generation module; 300 Mode selection module; 310 Local signaling service; 320 Local HTTP service; 410 WebRTC transmission link; 420 MSE media buffer transmission link; 430 Control backhaul channel; 431 Control data transmission unit. Detailed Implementation

[0024] I. System Architecture.

[0025] like Figure 1 As shown, this embodiment includes a mobile terminal transmitter 100 and a browser receiver 200. The mobile terminal transmitter 100 includes a screen / audio acquisition module 110, a coordinate mapping execution module 120, an encoding / encapsulation service module 130, a service discovery and pairing module 140, a restricted network access module 150, a mode selection module 300, a local signaling service 310, and a local HTTP service 320. The browser receiver 200 includes a browser runtime environment 201, which loads a browser receiving page 202. The browser receiving page 202 includes a capability detection module 210, a playback rendering module 220, a browser pre-mapping module 230, a unified control data generation module 240, and a control data sending unit 431.

[0026] WebRTC transmission link 410, MSE media buffer transmission link 420, and control return channel 430 are communication links or channels between the mobile terminal sending end 100 and the browser receiving page 202, and are not independent third-party entities. The mode selection module 300, local signaling service 310, and local HTTP service 320 are all located on the mobile terminal sending end 100 side.

[0027] II. Acquisition, Encoding, and Determination of Target Media Type.

[0028] After obtaining screen capture authorization, the screen / audio capture module 110 creates a virtual display and outputs the virtual display to the encoder input surface, enabling the screen image to enter the video encoding process. Video encoding parameters include resolution, frame rate, bit rate, keyframe interval, and bit rate control method.

[0029] The target video type is determined by the mobile terminal sending end 100 based on video encoding parameters and the video type support status reported by the browser receiving page 202. For example, in one implementation, the target video type is the H.264 video type supported by the browser runtime environment 201. When the system version, user authorization, audio capture strategy of the application being captured, and the same user configuration conditions meet the audio capture conditions, the mobile terminal sending end 100 captures internally played audio and encodes it into audio data. The target audio type is determined by the mobile terminal sending end 100 based on audio encoding parameters and the audio type support status reported by the browser receiving page 202.

[0030] III. Capability Testing and Dual-Mode Selection

[0031] like Figure 2 As shown, the mobile terminal sending end 100 provides the browser receiving page 202 to the browser receiving end 200. After the browser receiving page 202 is loaded, the capability detection module 210 detects the first capability set and the second capability set, and reports the capability vectors to the mobile terminal sending end 100. The first capability set includes the constructible state of RTCPeerConnection, the creatable state of RTCDataChannel, and the reachable state of the local signaling service 310. The second capability set includes the available state of MediaSource, the creatable state of SourceBuffer, and the target video type support state; in the embodiment of transmitting audio data, the second capability set also includes the target audio type support state.

[0032] The mode selection module 300 generates a target transmission link based on the capability vector and operating strategy parameters. The operating strategy parameters include a low-latency priority flag, a quality priority flag, a first-frame waiting threshold, a maximum buffer length, a target frame rate, a target bit rate, and a mode fallback order. If the first capability set meets the preset conditions and low latency is prioritized, the WebRTC transmission link 410 is selected; if the second capability set meets the preset conditions and quality is prioritized, or if the WebRTC transmission link 410 fails to establish, the MSE media buffer transmission link 420 is selected. In the event of a first-frame timeout, connection loss, or SourceBuffer append error, the browser receiving page 202 re-reports the capability vector, and the mode selection module 300 regenerates the target transmission link. Figure 2 The exception handling arrow in the code returns to the capability detection process, indicating a re-detection and rollback for reselection.

[0033] IV. WebRTC Transmission Link.

[0034] like Figure 3As shown, when the target transmission link is the WebRTC transmission link 410, the mobile terminal sending end 100 sends its local session description and connection candidate information to the browser receiving page 202 via the local signaling service 310, and receives the session description and connection candidate information returned by the browser receiving page 202. After the connection is established, the mobile terminal sending end 100 transmits video data to the playback rendering module 220 via the WebRTC transmission link 410. The control return channel 430 is bound to the RTCDataChannel, and the control data sending unit 431 sends unified control data to the mobile terminal sending end 100.

[0035] Under direct connection conditions within the same local area network (LAN), connection candidate information includes host candidates or LAN candidates. When remote negotiation conditions are met, the local signaling service 310 can cooperate with the remote security signaling service to conduct session negotiation.

[0036] V. MSE Media Buffer Transmission Link.

[0037] like Figure 4 As shown, when the target transmission link is the MSE media buffer transmission link 420, the encoding / encapsulation service module 130 encapsulates the video data into an initialization segment and a media segment with a segment sequence number and a media timestamp. The local HTTP service 320 provides the receiving page, segment address, and media segment to the browser receiving page 202. The playback rendering module 220 creates a MediaSource and a SourceBuffer, and appends the initialization segment and the media segment in sequence.

[0038] In the embodiment for transmitting audio data, the encoding / encapsulation service module 130 synchronously encapsulates the audio data and video data according to the media timestamp. The browser receiving page 202 maintains a media segment queue and a SourceBuffer append queue for each casting session, and performs one of the following processing actions based on the current buffer length: jumping to the edge of the live stream, speeding up the playback, discarding old segments, or pausing to request new segments. The control return channel 430 is bound to a WebSocket in this mode.

[0039] VI. Unified control data and reverse control.

[0040] The unified control data can be a binary frame or a structured text frame, and includes a version field, an event type field, a flag field, an event sequence number field, a session identifier field, a source frame horizontal coordinate field, a source frame vertical coordinate field, a source frame width field, a source frame height field, and a checksum field. The RTCDataChannel corresponding to the WebRTC transport link 410 and the WebSocket corresponding to the MSE media buffer transport link 420 carry the unified control data in the same format.

[0041] like Figure 5 As shown, the browser pre-mapping module 230 in page 202 reads the video element boundary, the source screen size, and the black border offset generated by the proportional display, and converts the pointer event position into source screen coordinates. Specifically, the smaller value is first divided by the width of the video element and the width of the source screen, and the smaller value is divided by the height of the video element and the height of the source screen, and then the horizontal and vertical black border offsets are calculated based on the proportional display ratio. The pointer event position is then subtracted from the top left corner of the video element and the corresponding black border offset, divided by the proportional display ratio, and the result is limited to the range of the source screen to obtain the source screen coordinates.

[0042] like Figure 6 As shown, after receiving unified control data, the coordinate mapping execution module 120 in the mobile terminal transmitter 100 verifies the session identifier, event sequence number, and verification fields. After successful verification, the coordinate mapping execution module 120 performs normalization and rotation transformations as follows: Let the source screen's horizontal coordinate be sx, the source screen's vertical coordinate be sy, the source screen's width be sw, the source screen's height be sh, the target screen's width be tw, the target screen's height be th, the target's horizontal coordinate be tx, and the target's vertical coordinate be ty. First, divide sx by sw to obtain the normalized horizontal coordinate nx, and divide sy by sh to obtain the normalized vertical coordinate ny. When the current rotation direction is 0 degrees, tx equals nx multiplied by tw, and ty equals ny multiplied by th. When the current rotation direction is 90 degrees, tx equals the result of subtracting ny multiplied by tw, and ty equals nx multiplied by th. When the current rotation direction is 180 degrees, tx equals the result of subtracting nx multiplied by tw, and ty equals the result of subtracting ny multiplied by th. When the current rotation direction is 270 degrees, tx equals ny multiplied by tw, and ty equals one minus nyx multiplied by th. Then, tx is limited to the horizontal range of the target screen, and ty is limited to the vertical range of the target screen. The target control coordinates are obtained based on the limited tx and ty. Through the above mapping relationship, the source screen coordinates can be converted into control coordinates corresponding to the current screen orientation of the mobile terminal in landscape, portrait, and rotation switching scenarios.

[0043] like Figure 7 As shown, when the mobile terminal transmitter 100 does not obtain system-level input event injection permission, the mobile terminal transmitter 100 performs clicks, long presses, or swipes through the accessibility service path. When the mobile terminal transmitter 100 has platform signature, pre-installation permissions, or system-granted input event injection permission, the mobile terminal transmitter 100 performs touch events or button events through the system input injection path; when the system input injection permission is not satisfied, the system input injection path is disabled and the user falls back to the accessibility service path.

[0044] VII. Receiving entry point discovery, pairing, and resource release.

[0045] like Figure 8 As shown, the service discovery and pairing module 140 provides a receiving entry point through one of the following methods: direct address access, local network service publication, or near-field pairing service publication. The receiving entry point can carry the service port, protocol version, session digest, and pairing code digest. The mobile terminal transmitter 100 initiates service publication when it detects a receiving page access, an active media session, or that the pairing is within its validity period. After a preset idle time, it stops service publication and releases the multicast lock, broadcast object, media segment queue, and control connection.

[0046] 8. Restricted network hotspot redirection access.

[0047] like Figure 9 As shown, when the browser receiving terminal 200's direct access to the private address of the mobile terminal sending terminal 100 is restricted, the restricted network access module 150 performs hotspot redirection access. The hotspot management unit 151 enables a wireless hotspot. After the browser receiving terminal 200 accesses the hotspot, it accesses a preset public address or domain name in the browser runtime environment 201. The access request identification unit 152 identifies the access request, and the local routing unit 153 routes the access request to the local receiving page service unit 154 through local DNS rewriting, transparent proxy, or virtual private network traffic splitting.

[0048] Local receiving page service unit 154 returns browser receiving page 202 and session parameters. After browser receiving page 202 loads, it proceeds... Figure 2 The dual-mode capability detection process is shown. The loop protection unit 155 is used to enable local HTTP service 320, local signaling service 310, or other local service connections to bypass the virtual private network traffic diversion in order to avoid local forwarding loops.

[0049] like Figure 10 As shown, the restricted network access module 150 is located within the mobile terminal transmitter 100, forming an access closed loop with the browser receiver 200, the local HTTP service 320, the MSE media buffer transmission link 420, and the control return channel 430. The restricted network access module 150 does not change the format of the unified control data, nor does it change the mode selection logic of the WebRTC transmission link 410 and the MSE media buffer transmission link 420.

[0050] IX. Anomaly Recovery and Security Verification.

[0051] The unified control data carries a session identifier, event sequence number, and verification field. The mobile terminal transmitter 100 refuses to execute unified control data that is expired, replayed, out of order exceeding the threshold, or fails verification. When the target transmission link fails to establish, the first media frame exceeds a preset waiting threshold, the RTCDataChannel is disconnected, the WebSocket is disconnected, or the SourceBuffer append error reaches a preset number, the browser receiving page 202 re-reports the capability vector, and the mode selection module 300 reselects the target transmission link. Because the semantics of the unified control data remain unchanged, the mobile terminal transmitter 100 does not need to change its control parsing and coordinate mapping logic after the target transmission link is switched.

Claims

1. A dual-mode browser screen projection and reverse control method executed by a mobile terminal sending end, characterized in that, include: After receiving the screen capture authorization, the mobile terminal transmitter establishes a virtual display connected to the encoder input surface and captures screen video frames, then encodes the screen video frames into video data. The mobile terminal sending end provides a browser receiving page to the browser receiving terminal, and the browser receiving terminal includes a browser runtime environment, and the browser runtime environment loads the browser receiving page. The mobile terminal receiving end receives the capability vector reported by the browser receiving page. The capability vector is used to characterize the availability of candidate media transmission links, including WebRTC transmission links and MSE media buffer transmission links. The mode selection module in the mobile terminal transmitter generates a target transmission link from the candidate media transmission links based on the capability vector and the operation strategy parameters. The mobile terminal transmitter establishes a corresponding media output channel according to the target transmission link, and binds the control return channel corresponding to the target transmission link to RTCDataChannel or WebSocket; The mobile terminal transmitting end transmits the video data to the browser receiving page through the target transmission link, and enables the RTCDataChannel or the WebSocket to carry unified control data of the same format; The mobile terminal receiving end receives the unified control data through the control feedback channel, and calls the input execution path to complete the reverse control based on the control information parsed from the unified control data.

2. The method according to claim 1, characterized in that, The capability vector includes a first capability set and a second capability set. The first capability set includes the constructible state of RTCPeerConnection, the creatable state of RTCDataChannel, and the reachable state of signaling services. The second capability set includes the available state of MediaSource, the creatable state of SourceBuffer, and the target video type support state. The target video type is determined by the mobile terminal sending end based on the video encoding parameters and the video type support status reported by the browser receiving page.

3. The method according to claim 1, characterized in that, When the system version, user authorization, audio capture strategy of the application being collected, and the same user configuration conditions meet the audio collection conditions, the mobile terminal transmitter collects the internally played audio and encodes it into audio data; the capability vector also includes the target audio type support status, the target audio type is determined by the mobile terminal transmitter based on the audio encoding parameters and the audio type support status reported by the browser receiving page; when the target audio type support status meets the preset conditions, the mobile terminal transmitter transmits the audio data and the video data synchronously.

4. The method according to claim 1, characterized in that, The operation strategy parameters include one or more of the following: low latency priority flag, image quality priority flag, first frame waiting threshold, maximum buffer length, target frame rate, target bit rate, and mode fallback order. The mode selection module generates the target transmission link in the order of priority mode capability satisfaction, priority mode successful establishment, and other mode fallback. When the target transmission link fails to be established, the first media frame exceeds the preset waiting threshold, the RTCDataChannel is disconnected, the WebSocket is disconnected, or the SourceBuffer append error reaches a preset number, the mobile terminal sending end receives the capability vector re-reported by the browser receiving page, and regenerates the target transmission link while maintaining the same session identifier or generating a derived session identifier.

5. The method according to claim 1, characterized in that, When the browser receiving terminal has restricted direct access to the private address of the mobile terminal sending terminal, the mobile terminal sending terminal performs a restricted network access step, which includes: Enable a wireless hotspot for the browser's receiving terminal to access; Identify the access request initiated by the browser receiving terminal to a preset public address or domain name; The access request is routed to the receiving page service running locally on the mobile terminal by means of local parsing rewriting, transparent proxy, or virtual private network traffic offloading. The receiving page service returns the browser receiving page and session parameters to the browser receiving terminal; The restricted network access step is executed by the restricted network access module in the mobile terminal sending end. The restricted network access module includes a hotspot management unit, an access request identification unit, a local routing unit, a local receiving page service unit, and a loop protection unit. The loop protection unit is used to enable local service connections to bypass the VPN traffic diversion.

6. The method according to claim 1, characterized in that, When the target transmission link is a WebRTC transmission link, the mobile terminal sending end sends local session description and connection candidate information to the browser receiving page via local signaling service, and receives session description and connection candidate information returned by the browser receiving page; under the condition of direct connection in the same local area network, the connection candidate information includes host candidate or local area network candidate. When the remote negotiation conditions are met, the signaling service is switched to the remote security signaling service.

7. The method according to claim 1, characterized in that, When the target transmission link is an MSE media buffer transmission link, the mobile terminal sending end encapsulates the video data into an initialization segment and a media segment with a segment sequence number and a media timestamp, and outputs it to the browser receiving page via a local HTTP service; the initialization segment includes file type information and media metadata information, and the media segment includes segment header information and media data information, and the segment header information carries the segment sequence number, decoding timestamp, and sample duration.

8. The method according to claim 1, characterized in that, The unified control data is a binary frame or a structured text frame. The unified control data includes a version field, an event type field, a flag field, an event sequence number field, a session identifier field, a source screen horizontal coordinate field, a source screen vertical coordinate field, a source screen width field, a source screen height field, and a verification field. The browser receiving page provided by the mobile terminal to the browser receiving terminal includes a pre-mapping instruction. When the pre-mapping instruction is executed in the browser's runtime environment, the browser receiving page reads the width, height, and top-left corner position of the video element on the page, reads the source screen width and height, calculates the proportional display ratio, display area size, and black border offset, subtracts the top-left corner position of the video element and the corresponding black border offset from the pointer event position, divides it by the proportional display ratio, and limits the result to the source screen range to generate the source screen horizontal and vertical coordinates in the unified control data.

9. The method according to claim 8, characterized in that, The mobile terminal receiving end receives continuous movement pointer events output by the browser receiving page after sampling, event merging, or sequential discarding, and combines continuous coordinate points between the same press and release events into sliding control data; after verifying the session identifier field, event sequence number field, and verification field, the mobile terminal receiving end obtains the current target screen width, target screen height, and current rotation direction, and performs the following coordinate transformations: divide the source screen horizontal coordinate by the source screen width to obtain the normalized horizontal coordinate, and divide the source screen vertical coordinate by the source screen height to obtain the normalized vertical coordinate; when the current rotation direction is 0 degrees, multiply the normalized horizontal coordinate by the target screen width to obtain the target horizontal coordinate, and multiply the normalized vertical coordinate by the target screen height to obtain the target vertical coordinate; when the current rotation direction is 90 degrees, multiply the result of subtracting the normalized vertical coordinate by the target screen width to obtain the target horizontal coordinate. The target vertical coordinate is obtained by multiplying the normalized horizontal coordinate by the target screen height; when the current rotation direction is 180 degrees, the target horizontal coordinate is obtained by multiplying the result of subtracting the normalized horizontal coordinate by the target screen width, and the target vertical coordinate is obtained by multiplying the result of subtracting the normalized vertical coordinate by the target screen height; when the current rotation direction is 270 degrees, the target horizontal coordinate is obtained by multiplying the normalized vertical coordinate by the target screen width, and the target vertical coordinate is obtained by multiplying the result of subtracting the normalized horizontal coordinate by the target screen height; the target horizontal coordinate is limited to the horizontal range of the target screen, and the target vertical coordinate is limited to the vertical range of the target screen, and the target control coordinate is obtained from the limited target horizontal coordinate and target vertical coordinate; when the mobile terminal sending end does not obtain system-level input event injection permission, the mobile terminal sending end performs click, long press or swipe through the accessibility service path; When the mobile terminal sending end has platform signature, pre-installation permissions, or system-granted input event injection permissions, the mobile terminal sending end executes touch events or key events through the system input injection path.

10. A mobile terminal transmitter, characterized in that, The device includes a processor, a memory, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it causes the mobile terminal transmitter to perform the method described in any one of claims 1 to 9.

Images