Smartphone assisted control system, method, medium and program product

Abstract

The application provides an auxiliary control system, method, medium and program product of a smartphone, comprising: an acquisition module acquires current display interface data of the smartphone; an FPGA module analyzes and processes the current display interface data to obtain image data; a processing module performs interface information extraction and semantic analysis processing on the received image data based on a visual large model, identifies interface elements in the image data, and generates interface interaction information according to the interface elements and a target operation task; an interaction module executes a first interaction mode or a second interaction mode according to the generated interface interaction information; wherein in the first interaction mode, the interface interaction information is output and displayed to prompt a user to perform corresponding interface operations on the smartphone; in the second interaction mode, input control instructions are generated according to the interface interaction information to control the smartphone to perform corresponding interface operations. The application can avoid privacy leakage and data abuse risks, greatly improving privacy security.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an auxiliary control system, method, medium, and program product for a smartphone. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology, AI agents with multimodal perception and autonomous decision-making capabilities have begun to be deeply integrated into mobile terminals, forming intelligent assistants capable of taking over the phone screen and performing complex tasks. Taking Doubao Mobile Assistant as an example, it achieves cross-app operation functionality through an automated approach using a graphical user interface (GUI) that combines visual recognition with simulated touch. Specifically, through deep cooperation with mobile terminal manufacturers, it obtains high-level system-level permissions, such as the INJECT_EVENTS permission, enabling the AI ​​assistant to view screen content in real time like a natural person and inject input commands into the operating system through simulated clicks, swipes, and other touch events, thereby completing cross-platform operations for various applications.

[0003] However, existing AI assistant solutions require vague and broad system-level permissions, allowing them to indiscriminately read all information displayed on the screen, including users' personal privacy data, financial account details, instant messaging content, and sensitive corporate information. This permission configuration breaches the security boundaries established by traditional application sandbox isolation mechanisms, resulting in a lack of effective local isolation and granular access control for screen data during transmission to cloud-based large-scale models for processing. This poses serious risks of privacy leaks and data misuse.

[0004] Therefore, it is necessary to provide an auxiliary control system, method, medium, and program product for smartphones to solve the aforementioned problems existing in the prior art. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide an auxiliary control system, method, medium and program product for smartphones, which solves the technical problem that the vague and broad system-level permissions in the prior art lead to serious privacy leakage risks and data abuse risks for user privacy data.

[0006] To achieve the above and other related objectives, a first aspect of this application provides an auxiliary control system for a smartphone, comprising: an acquisition module for acquiring current display interface data of the smartphone; an FPGA module for parsing and processing the current display interface data to obtain image data corresponding to the current display interface of the smartphone; a processing module for extracting interface information and performing semantic parsing processing on the received image data based on a large visual model to identify interface elements in the image data, and generating interface interaction information based on the identified interface elements and the target operation task; and an interaction module for executing a first interaction mode or a second interaction mode based on the generated interface interaction information; wherein, in the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone; and in the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform corresponding interface operations.

[0007] In some embodiments of the first aspect of this application, the processing module includes an extraction unit, a parsing unit, and an information generation unit; wherein, the extraction unit is used to perform feature extraction and text recognition on the received image data to obtain interface information corresponding to the image data; the parsing unit is used to perform semantic parsing based on the extracted interface information to obtain interface elements corresponding to the interface information; and the information generation unit is used to generate interface interaction information based on the interface elements and the target operation task.

[0008] In some embodiments of the first aspect of this application, the FPGA module includes a decoding unit and a decomposition unit; wherein the decoding unit is used to decode the acquired current display interface data; and the decomposition unit is used to decompose the decoded current display interface data into image data in image format.

[0009] In some embodiments of the first aspect of this application, the FPGA module further includes an encoding unit for encoding the received interface interaction information to convert it into a data format for output display.

[0010] In some embodiments of the first aspect of this application, a display output interface is also included, which is used to output and display the encoded interface interaction information on a display device to prompt the user to perform corresponding interface operations on the smartphone.

[0011] In some embodiments of the first aspect of this application, the interaction module includes a bypass unit and a wireless communication unit; wherein, the bypass unit is used to bypass the processing module, receive manual operation signals generated by the user through an external input device, and convert and process the manual operation signals to generate corresponding input control data; the wireless communication unit is used to send the input control data to a smartphone, so that the smartphone responds to the input control data to perform corresponding interface operations.

[0012] In some embodiments of the first aspect of this application, the interaction module further includes a simulated input unit, configured to generate corresponding input control instructions based on the received interface interaction information, and send the input control instructions to a smartphone via a wireless communication unit, so that the smartphone responds to the input control instructions and performs corresponding interface operations.

[0013] To achieve the above and other related objectives, a second aspect of this application provides an auxiliary control method for a smartphone, comprising: acquiring current display interface data of the smartphone; parsing the current display interface data to obtain image data corresponding to the current display interface of the smartphone; extracting interface information and performing semantic parsing on the received image data based on a large visual model to identify interface elements in the image data, and generating interface interaction information based on the identified interface elements and the target operation task; executing a first interaction mode or a second interaction mode based on the generated interface interaction information; wherein, in the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone; in the second interaction mode, an input control command is generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform corresponding interface operations.

[0014] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0015] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the method.

[0016] As described above, the smartphone auxiliary control system, method, medium, and program product of this application have the following beneficial effects:

[0017] The application first acquires the current display interface data of the smartphone and transmits it to the FPGA module. Then, the FPGA module parses the acquired display interface data to obtain image data corresponding to the current display interface of the smartphone and transmits the image data to the processing module. Next, the processing module extracts interface information and performs semantic parsing on the received image data based on a large visual model to identify interface elements in the image data and generates interface interaction information based on the identified interface elements and the target operation task. Finally, the interaction module executes either a first interaction mode or a second interaction mode based on the generated interface interaction information. In the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone. In the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform corresponding interface operations. This application achieves secure and controllable operation of the smartphone without breaking the original sandbox isolation mechanism of traditional applications or relying on high-level system injection permissions, thereby avoiding the risks of privacy leakage and data abuse and greatly improving privacy security. Attached Figure Description

[0018] Figure 1 The diagram shown is a block diagram of an auxiliary control system for a smartphone according to an embodiment of this application.

[0019] Figure 2 The diagram shown illustrates the working principle of an auxiliary control system for a smartphone according to an embodiment of this application.

[0020] Figure 3 The diagram shown is a flowchart illustrating an assisted control method for a smartphone according to an embodiment of this application. Detailed Implementation

[0021] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0022] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0023] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0025] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0026] <1> Vision Large Models (VLMs) are foundational models trained on large-scale images and multimodal data, enabling cross-task and cross-modal visual understanding, generation, and reasoning capabilities.

[0027] <2> FPGA (Field Programmable Gate Array) module: a hardware chip that can be programmed to perform specific functions and is widely used in high-performance computing and real-time data processing.

[0028] <2> USB (Universal Serial Bus) interface: It is an industry-standard serial bus and input / output interface technology. Its core features include support for hot-swapping and plug-and-play, allowing users to safely connect or disconnect peripherals while the device is running without restarting the system. In addition, the USB interface can provide power to the connected device while transmitting data. This core function makes it widely used in various information and communication products such as personal computers, mobile devices, photographic equipment, digital televisions, and game consoles.

[0029] <3> SDIO (Secure Digital Input and Output) interface: This technology is developed based on the SD card interface. It is compatible with traditional SD cards and supports connection to various external devices, such as Bluetooth, Wi-Fi, and GPS. It realizes data transmission through the SD bus and commands, and not only has storage functions, but can also expand other functions.

[0030] <4> BLE (Bluetooth Low Energy) technology is a short-range, low-cost, interoperable wireless technology that utilizes many intelligent methods to minimize power consumption. BLE's operating mode is ideal for transmitting data from miniature wireless sensors or other peripherals using fully asynchronous communication, such as remote controls.

[0031] <5> HDMI (High-Definition Multimedia Interface) is a fully digital video and audio transmission interface that can transmit uncompressed audio and video signals simultaneously. Its emergence has greatly simplified the connection between devices and replaced traditional analog signal interfaces such as SCART and RCA.

[0032] <6> Type-C interface: is a small, reversible USB interface standard that supports high-speed data transmission and high-power charging.

[0033] <7> PCIe (Peripheral Component Interconnect Express) interface: is a high-speed serial computer expansion bus standard used to connect computer motherboards to various external devices.

[0034] To facilitate understanding of the embodiments of this application, in conjunction with Figure 1 and Figure 2 Detailed explanation. Figure 1 A block diagram of an auxiliary control system for a smartphone according to an embodiment of the present invention is shown. Figure 2 This illustration shows a schematic diagram of the working principle of an auxiliary control system for a smartphone according to an embodiment of the present invention. The auxiliary control system 100 of the smartphone in this embodiment includes: an acquisition module 101, an FPGA module 102, a processing module 103, and an interaction module 104. The acquisition module 101 is connected to the FPGA module 102, the FPGA module 102 is connected to the processing module 103, and the processing module 103 is connected to the interaction module 104, forming a processing chain of current display interface data acquisition - image parsing - visual large model semantic understanding - interface interaction execution.

[0035] The acquisition module 101 is used to acquire the current display interface data of the smartphone.

[0036] For example, the current display interface data is the screen output data corresponding to the current display interface of the smartphone, such as the current display interface data being output in the form of a video stream.

[0037] In some embodiments of this application, the smartphone's auxiliary control system further includes a Type-C interface for communicating with the smartphone to obtain the smartphone's current display interface data.

[0038] By using an external hardware-type Type-C interface, the current display data of the smartphone is obtained in real time. This moves the action of "looking at the screen" from inside the smartphone system to external hardware independent of the smartphone. This isolates the smartphone's auxiliary control system from the internal data of the smartphone, eliminating the need for the smartphone to grant system-level screen reading permissions. This avoids the privacy risks caused by the abuse of system-level permissions from the data input path.

[0039] Specifically, the module communicates with a smartphone via a Type-C interface and receives the current display interface data output by the smartphone. That is, the smartphone outputs the real-time display content presented on the current display interface through the display output channel of the Type-C interface. The acquisition module collects the output current display interface data and transmits the collected current display interface data to the FPGA module as the raw data for subsequent image parsing and interface understanding.

[0040] FPGA module 102 is used to parse and process the data of the current display interface to obtain image data corresponding to the current display interface of the smartphone.

[0041] The FPGA module performs low-latency hardware-level parsing and processing on the received current display interface data, realizing low-latency, high-throughput conversion of the current display interface data of the smartphone from video stream to image data, providing real-time, complete and uniformly formatted input data for subsequent interface recognition of large visual models.

[0042] In some embodiments of this application, the FPGA module includes a decoding unit and a decomposition unit; wherein, the decoding unit is used to decode the acquired current display interface data; and the decomposition unit is used to decompose the decoded current display interface data into image data in image format.

[0043] By having the decoding and decomposition units work in parallel at the hardware level, the video stream output by the smartphone is converted into image data in image format frame by frame in an orderly manner, providing continuous and frame-free input for the interface understanding of the large visual model, thus balancing real-time processing and interface integrity.

[0044] Specifically, the decoding unit, for example, is a video decoder, which decodes the acquired data of the current display interface transmitted in the form of a video stream to reconstruct frame-by-frame image data. The decomposition unit decomposes the decoded frame-by-frame image data into image data in picture format and transmits it to the processing module in frame order through the communication interface. The communication interface uses a USB or SDIO interface to transmit the image data obtained by the decomposition unit to the processing module. Alternatively, the communication interface can also be implemented using interface protocols such as PCIe (Peripheral Component Interconnect Express), UART (Universal Asynchronous Receiver / Transmitter), I2C (Inter-Integrated Circuit), or SPI (Serial Peripheral Interface).

[0045] The processing module 103 is used to extract interface information and perform semantic parsing processing on the received image data based on the visual large model, so as to identify interface elements in the image data and generate interface interaction information based on the identified interface elements and the target operation task.

[0046] By leveraging the cross-scene visual semantic understanding capabilities of large visual models, smartphone interface images are transformed into structured and operable interface element semantics. Combined with the target operation task, interface interaction information is generated, enabling the system to understand screen content and user intent across applications and interfaces, providing accurate decision-making basis for subsequent interaction execution.

[0047] In some embodiments of this application, the processing module includes an extraction unit, a parsing unit, and an information generation unit; wherein, the extraction unit is used to perform feature extraction and text recognition on the received image data to obtain interface information corresponding to the image data; the parsing unit is used to perform semantic parsing based on the extracted interface information to obtain interface elements corresponding to the interface information; and the information generation unit is used to generate interface interaction information based on the interface elements and the target operation task.

[0048] By designing the processing module into an extraction unit, a parsing unit, and an information generation unit, the extraction, parsing, and interactive information generation of interface information are realized respectively. Each unit can be implemented independently and optimized separately, reducing the complexity of a single model completing the entire process of perception, understanding, and decision-making, and improving the accuracy, maintainability, and scalability of interface recognition and interactive information generation.

[0049] Specifically, the extraction unit first performs visual feature extraction and text recognition on the received image data. For example, it uses a visual feature extraction network to extract visual features such as control outlines, icons, and layouts from the image, and combines this with Optical Character Recognition (OCR) to recognize the text content in the image, thus obtaining the interface information corresponding to the image data. Then, the parsing unit performs semantic parsing on the interface information, determining the type and function of each object based on the meaning of the interface text, the shape of the controls, and the spatial layout, obtaining interface elements with operational semantics, such as buttons, menus, input boxes, and icons. Finally, the information generation unit matches the interface elements with the target operation task, determining the interface elements and operation methods required to complete the task, and generating the corresponding interface interaction information. It should be noted that the extraction unit, parsing unit, and information generation unit respectively implement visual perception, semantic understanding, and task decision-making, and can be deployed on the same AI chip or placed in different computing units.

[0050] The interaction module 104 is used to execute a first interaction mode or a second interaction mode based on the generated interface interaction information. In the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone. In the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform corresponding interface operations.

[0051] By flexibly switching between the first and second interaction modes, it supports both manual assistance with prompts and user-executed actions, and autonomous operation of the phone by simulating peripheral devices. Neither mode relies on the phone's system-level event injection permissions, thus avoiding the AI ​​operation being identified as a cheat and subject to platform retaliation. It also allows users to manually take over the phone operation when necessary.

[0052] Specifically, the first interaction mode is the AI ​​prompt and user execution mode, where the system only presents operation prompts to the user without directly controlling the phone interface. In this mode, the final control is retained by the user, with the AI ​​only providing operation prompts and the user deciding whether and how to execute them, thus balancing automation efficiency, user control, and the responsibility boundary for the consequences of the operation. The second interaction mode is the AI ​​agent autonomous execution mode, where the system directly drives the smartphone to execute corresponding interface operations in the form of input control commands. In this mode, the system sends input control commands to the smartphone via a wireless communication link, simulating a peripheral device. The smartphone only treats the commands as ordinary peripheral inputs and responds accordingly, without needing to grant system-level event injection permissions, thereby avoiding the risk of platform countermeasures.

[0053] In some embodiments of this application, the FPGA module further includes an encoding unit for encoding the received interface interaction information to convert it into a data format for output display.

[0054] By designing an encoding unit, the processed interface interaction information is converted into a standard display format for output, forming the operation prompts and display output links in the user execution closed loop under the first interaction mode, enabling the operation prompts to be presented to the user with low latency outside the smartphone.

[0055] Specifically, the encoding unit, for example, is a video encoder, which receives the interface interaction information generated and transmitted back by the information generation unit, performs video encoding processing on the interface interaction information, including operation prompts, and converts it into an output display data format that can be directly received by an external display device, thereby supporting the presentation of AI processing results (including operation prompts) on the display device.

[0056] In some embodiments of this application, a display output interface is also included, which is used to output and display the encoded interface interaction information on a display device to prompt the user to perform corresponding interface operations on the smartphone.

[0057] By utilizing a display output interface independent of the smartphone screen, interactive information including operation prompts is presented on an external display device. Users can operate the smartphone while viewing the operation prompts, with no interference between display and operation, thus improving the intuitiveness and operational efficiency of human-computer collaboration.

[0058] Specifically, the display output interface is connected to the encoding unit of the FPGA module, outputting the encoded interface interaction information to the display device for display. The display output interface uses an HDMI interface, but can also use DP (DisplayPort), eDP (Embedded DisplayPort), LVDS (Low Voltage Differential Signaling), VGA (Video Graphics Array), or other display output interfaces.

[0059] For example, the display device is independent of the smartphone settings, such as an external monitor. In the first interaction mode, the interface interaction information containing operation prompts is encoded and presented to the display device through the display output interface. After viewing the operation prompts, the user performs the corresponding interface operation on the smartphone.

[0060] In some embodiments of this application, the interaction module includes a bypass unit and a wireless communication unit; wherein, the bypass unit is used to bypass the processing module, receive manual operation signals generated by the user through an external input device, and convert and process the manual operation signals to generate corresponding input control data; the wireless communication unit is used to send the input control data to a smartphone, so that the smartphone responds to the input control data to perform corresponding interface operations.

[0061] By designing a bypass unit, a direct manual control channel is provided that bypasses the processing module. Users can directly operate the smartphone at any time using an external input device. In the first interaction mode, when the user performs the operation, it does not need to go through the AI ​​processing link, reducing response latency, while ensuring the user's ultimate control over the operation of the phone.

[0062] Specifically, the bypass unit, for example, is a bypass circuit. In the first interaction mode, when the user directly operates the smartphone manually, the interface interaction information generated by the information generation unit is transmitted to the encoding unit. After processing by the encoding unit, it is displayed on an external display device via the display output interface. The user generates a manual operation signal through an external input device (e.g., a mouse). This manual operation signal is received and converted by the bypass circuit to generate corresponding input control data. This input control data does not pass through the AI ​​processing link of the processing module. The wireless communication unit uses WiFi or BLE technology to send the input control data to the smartphone via the wireless communication link, enabling the smartphone to respond to the input control data and execute the corresponding interface operation, thereby realizing AI prompts and user execution. In addition, the wireless communication link can also be implemented using wireless transmission protocols such as Bluetooth and Zigbee.

[0063] In some embodiments of this application, the interaction module further includes a simulated input unit, which generates corresponding input control commands based on the received interface interaction information, and sends the input control commands to a smartphone via a wireless communication unit, so that the smartphone responds to the input control commands and performs corresponding interface operations.

[0064] In existing technologies, AI assistants simulate “non-human” operations such as clicking and swiping in a programmatic manner. Such operations are easily identified by various applications as abnormal operations or plug-in attacks, thereby triggering the application platform’s “self-defense” mechanism and causing the AI ​​assistant to be banned from use by the platform.

[0065] By designing a simulated input unit, the interface interaction information is converted into standard input events that can be recognized by the smartphone in the form of a simulated peripheral device. This allows for direct operation of the smartphone interface without requiring the smartphone to grant system-level event injection permissions. Moreover, the operation behavior appears as ordinary peripheral device input on the application side, which can effectively avoid the problem of "non-natural person" operations being identified as abnormal or cheating and thus subject to platform retaliation.

[0066] Specifically, the simulated input unit is, for example, a simulated mouse. In the second interaction mode, the simulated mouse receives interface interaction information generated by the information generation unit, and converts the interface elements and operation methods indicated by the interface interaction information into corresponding input control commands, such as mouse clicks and swipes. The simulated mouse establishes a connection with the smartphone via a wireless communication link, enabling the smartphone to recognize the input control commands as input events from peripherals such as the mouse and execute corresponding interface operations, thereby realizing direct control of the smartphone interface by the AI ​​agent.

[0067] The beneficial effects of the smartphone auxiliary control system provided in this application embodiment are as follows: (1) Privacy and security: The system obtains the current display interface data in a hardware manner independent of the smartphone. The AI ​​processing is located outside the smartphone and does not need to obtain high-level permissions at the system level. This avoids the risk of privacy leakage and data abuse caused by excessive permissions and blurred boundaries breaking through the sandbox isolation mechanism. At the same time, it retains the AI ​​visual understanding and automatic operation capabilities and solves the privacy problems faced by existing AI assistants when directly installed on the phone; (2) Avoiding platform countermeasures: In the second interaction mode, the system sends input control commands to the smartphone through a wireless communication link as a simulated peripheral device. The operation behavior is manifested as ordinary peripheral input on the application side, rather than "non-natural person" injection operation in the system, thereby avoiding the AI ​​assistant being identified as abnormal or plug-in attack by the application and being banned by the platform. (3) Human-machine collaboration, controllable and reliable: The system provides two modes, the first interaction mode and the second interaction mode, and sets up a bypass channel to bypass the processing module. Users can either operate manually according to the AI ​​prompts or take over the mobile phone directly at any time using external devices such as a mouse, taking into account the automation efficiency, user control and the responsibility boundary of the operation consequences; (4) Low latency and high real-time: Utilizing the video decoding, image decomposition and video encoding capabilities of the FPGA module, the acquisition, format conversion and output of the processing results of the smartphone display interface are all completed in a hardware-level parallel pipeline manner, with low processing latency and high throughput, which can meet the needs of real-time auxiliary operation; (5) Strong cross-application generalization ability: The system realizes the extraction of interface information, semantic parsing and interactive decision-making based on the visual big model, and can understand the screen content and generate interface interaction information across applications and interfaces, and has a strong generalization ability for dynamic layout and different application interfaces.

[0068] Figure 3 This is a flowchart illustrating the assisted operation method for a smartphone provided in an embodiment of this application. Figure 3 As shown, the assisted operation method of a smartphone includes the following steps:

[0069] Step S31: Obtain the current display interface data of the smartphone;

[0070] Step S32: Parse and process the current display interface data to obtain image data corresponding to the current display interface of the smartphone;

[0071] Step S33: Based on the visual big model, extract interface information and perform semantic parsing on the received image data to identify interface elements in the image data, and generate interface interaction information based on the identified interface elements and target operation task.

[0072] Step S34: Execute the first interaction mode or the second interaction mode according to the generated interface interaction information; wherein, in the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform the corresponding interface operation on the smartphone; in the second interaction mode, input control commands are generated according to the interface interaction information and sent to the smartphone to control the smartphone to perform the corresponding interface operation.

[0073] It should be understood that the smartphone auxiliary control method provided in this application embodiment can realize the functions of the smartphone auxiliary control system described above, and will not be described in detail here for the sake of brevity.

[0074] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0075] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 3 The method in the illustrated embodiment.

[0076] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 3 The method in the illustrated embodiment.

[0077] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0078] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0079] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0083] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0084] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] In summary, addressing the technical problem of ambiguous and broad system-level permissions in existing technologies leading to serious privacy risks and data misuse hazards for user data, this application provides an auxiliary control system, method, medium, and program product for smartphones. The system first acquires the current display interface data of the smartphone via an acquisition module and transmits this data to an FPGA module. Then, the FPGA module parses and processes the acquired current display interface data to obtain image data corresponding to the current display interface of the smartphone, and transmits the image data to a processing module. Finally, the processing module performs interface information extraction and semantic parsing processing on the received image data based on a large visual model to identify boundaries in the image data. The application identifies interface elements and generates interface interaction information based on the identified interface elements and the target operation task. Finally, the interaction module executes either a first interaction mode or a second interaction mode based on the generated interface interaction information. In the first interaction mode, the interface interaction information is displayed to prompt the user to perform corresponding interface operations on the smartphone. In the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control it to perform corresponding interface operations. This application achieves secure and controllable operation of the smartphone without breaking the original sandbox isolation mechanism of traditional applications or relying on high-level system injection permissions, thereby avoiding the risks of privacy leakage and data abuse and greatly improving privacy security. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0086] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An auxiliary control system for a smartphone, characterized in that, include: The acquisition module is used to acquire the current display interface data of the smartphone; The FPGA module is used to parse and process the data of the current display interface to obtain image data corresponding to the current display interface of the smartphone. The processing module is used to extract interface information and perform semantic parsing on the received image data based on the visual big model, so as to identify interface elements in the image data and generate interface interaction information based on the identified interface elements and target operation tasks. The interaction module is used to execute a first interaction mode or a second interaction mode based on the generated interface interaction information; wherein, in the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone. In the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform the corresponding interface operation.

2. The auxiliary control system for a smartphone according to claim 1, characterized in that, The processing module includes an extraction unit, a parsing unit, and an information generation unit; wherein, the extraction unit is used to perform feature extraction and text recognition on the received image data to obtain interface information corresponding to the image data; the parsing unit is used to perform semantic parsing based on the extracted interface information to obtain interface elements corresponding to the interface information; and the information generation unit is used to generate interface interaction information based on the interface elements and the target operation task.

3. The auxiliary control system for a smartphone according to claim 1, characterized in that, The FPGA module includes a decoding unit and a decomposition unit; wherein, the decoding unit is used to decode the acquired current display interface data; and the decomposition unit is used to decompose the decoded current display interface data into image data in image format.

4. The auxiliary control system for a smartphone according to claim 1, characterized in that, The FPGA module also includes an encoding unit for encoding the received interface interaction information to convert it into a data format for output display.

5. The auxiliary control system for a smartphone according to claim 4, characterized in that, It also includes a display output interface, which is used to output the encoded interface interaction information to a display device to prompt the user to perform corresponding interface operations on the smartphone.

6. The auxiliary control system for a smartphone according to claim 1, characterized in that, The interaction module includes a bypass unit and a wireless communication unit; wherein, the bypass unit is used to bypass the processing module, receive manual operation signals generated by the user through an external input device, and convert and process the manual operation signals to generate corresponding input control data; the wireless communication unit is used to send the input control data to a smartphone, so that the smartphone responds to the input control data and executes the corresponding interface operation.

7. The auxiliary control system for a smartphone according to claim 6, characterized in that, The interaction module also includes a simulated input unit, which generates corresponding input control commands based on the received interface interaction information, and sends the input control commands to the smartphone via a wireless communication unit, so that the smartphone responds to the input control commands and performs corresponding interface operations.

8. A method for assisting in the operation of a smartphone, characterized in that, include: Obtain the current display screen data of the smartphone; The data of the current display interface is parsed and processed to obtain image data corresponding to the current display interface of the smartphone; Based on a large visual model, the received image data is processed to extract interface information and perform semantic parsing to identify interface elements in the image data, and to generate interface interaction information based on the identified interface elements and the target operation task. The first interaction mode or the second interaction mode is executed based on the generated interface interaction information; wherein, in the first interaction mode, the interface interaction information is output and displayed to prompt the user to perform corresponding interface operations on the smartphone. In the second interaction mode, input control commands are generated based on the interface interaction information and sent to the smartphone to control the smartphone to perform the corresponding interface operation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in claim 8.

10. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in claim 8.

Images