A car-machine application method and system based on screen recognition and instruction reconstruction

Abstract

The application provides a vehicle-machine application method and system based on screen recognition and instruction reconstruction; including: receiving natural language operation instructions of a vehicle driver and preprocessing; combining the current position of the vehicle, the operation period and the industrial operation task context, using multi-level matching rules, named entity recognition models and local instruction templates to perform business multi-level analysis and instruction reconstruction on the instructions, identify application identifiers, instruction types, control parameters and business priorities, and construct structured business instruction objects; sending the instructions through at least two communication channels with different protocol characteristics and according to the channel health state and business priority; the vehicle-machine end selects an online or offline mode according to the network state, determines the control mode according to the analyzed instruction type, and controls the application in a multi-modal way of URI Scheme, broadcast, business message and OCR screen recognition, dynamic coordinate scaling and barrier-free service simulated clicking.

Description

Technical Field

[0001] This invention relates to the field of intelligent control system technology for mobile communication interaction in vehicle systems, and more specifically, to a vehicle system application method and system based on screen recognition and command reconstruction. Background Technology

[0002] Currently, with the rapid development of intelligent connected vehicles, vehicle-mounted systems have become a core component of automotive intelligence. Especially in mining / logistics industrial vehicle-mounted systems, which are typically used in areas with weak signals and complex network environments, there are three unique pain points: weak network coverage and interrupted command transmission in mining areas; lack of unified APIs for non-standard apps; and the requirement that all industrial operation commands must be delivered. Insufficient public network coverage, unstable network quality, and frequent base station switching in mining or logistics operation areas can easily lead to interruptions or untimely delivery of operation commands. Furthermore, the lack of unified control interfaces for some non-standard business applications affects the unified control of application functions by the vehicle-mounted system. Existing technology US7398209B2 discloses a scheme for receiving natural language queries and executing commands, which... This solution achieves functionality through voice-based information query and command environments; however, it lacks the ability to reconstruct deep functionalities within specific applications when dealing with complex vertical industry business logic. Existing technology US20140249822A1 discloses a method for handling multimodal device interaction, determining intent by extracting non-voice interaction context; however, this solution cannot maintain the continuity of intent recognition in offline network environments. Existing technology US8214100B2 discloses a method for monitoring vehicles via wireless devices; however, this solution uses a single communication link, resulting in poor reliability in complex network environments.

[0003] Currently, with the development of intelligent industrial vehicles in mining transportation, port logistics, industrial park logistics and trunk logistics, vehicle-mounted systems have become an important part of industrial vehicle operation scheduling, task execution, navigation linkage, attendance tracking and status reporting. In the mining / logistics industrial vehicle-mounted control system (V2X) scenario, the V2X system has at least the following pain points: First, mining areas or logistics operation areas may experience situations such as the absence of public networks, weak networks, and frequent base station switching, which can easily cause work instructions to be interrupted or unable to be delivered; Second, there are a large number of non-standard business apps in the mining transportation, logistics scheduling, and freight management systems, with inconsistent interface standards and some functions not having open APIs, resulting in control blind spots in traditional interface calling methods; Third, industrial work instructions have high requirements for delivery rate and execution certainty. For example, if instructions such as work check-in, loading and unloading confirmation, and task change confirmation are not delivered or executed, it may lead to abnormal work processes or incorrect business data. Existing fixed command word solutions require drivers to memorize fixed formats, resulting in high learning costs and poor scalability; single communication channels lack reliability in the weak network environment of mines; control methods relying solely on application APIs cannot cover non-standard business apps without public interfaces or closed interfaces; and cloud-based solutions cannot provide continuous service when the network is unavailable. Therefore, an intelligent control method is needed that can adapt to the vehicle-mounted machine scenarios in mining / logistics industries, and can still complete the parsing, reliable transmission, and multimodal execution of work instructions even in weak network or network outage conditions. Existing technologies still have significant shortcomings in intelligent control of in-vehicle applications, mainly reflected in: limited natural language understanding capabilities, complex cross-application operations, poor control reliability when the network is unstable, and insufficient security. There are obvious drawbacks or shortcomings in the existing publicly available related technologies: (1) Existing technologies mostly use fixed command words or template matching to achieve vehicle control; Users must memorize fixed command formats, resulting in high learning costs; it cannot handle the diversity and ambiguity of natural language; and its command extensibility is poor, requiring updates to the command dictionary to add new features. (2) Existing technologies mostly use a single MQTT or WebSocket channel for remote control; when the network fluctuates or the server fails, control commands cannot be delivered; there is a lack of channel health monitoring and automatic switching mechanisms; and the optimal communication method cannot be dynamically selected according to network conditions. (3) Existing technologies mostly rely on specific APIs provided by applications for control. Each application is controlled independently, lacking a unified control framework; Different applications have inconsistent APIs, resulting in a large workload for development and adaptation; some functions lack API support (such as map zooming, view switching, etc.), making it impossible to achieve full control; application version updates may lead to API changes, resulting in high maintenance costs. (4) Existing technologies mostly place natural language understanding in the cloud for processing; the vehicle-mounted system only serves as the instruction execution terminal; Voice control is unavailable when the network is offline; cloud processing delays affect user experience; and uploading user privacy data to the cloud poses security risks. Existing technologies heavily rely on cloud-based NLP processing. In environments with weak or no network signals, such as mines and remote mountainous areas, voice commands cannot be parsed and executed, severely impacting the continuity of production operations.

[0004] Existing technologies typically only allow control of applications through standard APIs. For third-party applications that lack public APIs or have incomplete API functionality, as well as non-standard vertical industry apps, deep functional control cannot be achieved.

[0005] Existing remote control solutions mostly employ a single communication channel, which is prone to command loss due to network fluctuations when vehicles are moving at high speeds or when network environments are switching complexly. Their technical limitations lie in the risk of command accumulation or loss during the transient process of long-term connection disconnection and reconnection, especially with single protocols (such as MQTT). In high real-time scenarios such as freight dispatching, command delays or loss can lead to dispatch command failures, impacting logistics efficiency and creating safety hazards. Therefore, it is necessary to propose a vehicle-mounted system application method and system based on screen recognition and command reconstruction, involving remote intelligent control of vehicle-mounted systems (onboard Android systems), particularly suitable for scenarios involving the linkage between vehicle-mounted systems and business systems in vertical industries such as mining transportation and freight logistics. The technical solution includes natural language command parsing, multimodal communication channel management, application intelligent control, business system integration, and secure communication technologies to at least partially solve the problems existing in current technologies. Summary of the Invention

[0006] The summary of this invention introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary of this invention does not mean that it attempts to limit the key features and essential technical features of the claimed technical solution, nor does it mean that it attempts to determine the scope of protection of the claimed technical solution.

[0007] To at least partially solve the above problems, the present invention provides a vehicle infotainment application method based on screen recognition and command reconstruction, comprising: Based on network connection status monitoring results, the system automatically switches between online mode based on cloud parsing and offline mode based on local command templates. The received natural language commands are parsed at multiple levels based on preset matching rules to identify the target application identifier, command type and control parameters; Maintain at least two communication channels with different protocol characteristics, and dynamically select the target channel to transmit the parsed instructions based on the real-time monitored channel health status; Based on the target application's support for the control interface, the target control mode is selected from multiple preset control modes to execute the instruction; wherein, the multiple preset control modes include simulated interactive operations based on system auxiliary functions as the underlying control.

[0008] To address the issue that existing technologies heavily rely on cloud processing, resulting in limited availability in weak network environments, the proposed method is executed by the vehicle-mounted terminal processor, which invokes the microphone, GPS positioning module, vehicle communication module, and display screen. The method includes: Based on the network connection status monitoring results obtained by the vehicle communication module, the system automatically switches between an online mode based on cloud parsing and an offline mode based on local command templates. Based on preset matching rules, vehicle current location, work period and industrial work task context, the received natural language work instructions are parsed and reconstructed at multiple business levels to identify target application identifiers, instruction types, control parameters and business priorities, and generate structured business control instructions. Maintain at least two primary and backup communication channels with different protocol characteristics, and dynamically select one or more target communication channels to transmit the structured service control instructions based on the real-time monitored channel health status and the service priority. The control mode is determined based on the parsed instruction type, and the target control method is selected from multiple preset control modes to execute the structured business control instruction based on the target application's support for the control interface. Among them, the multiple preset control modes include simulated interactive operations based on screen recognition and system auxiliary functions as the underlying control, and there is an associated mapping relationship between the instruction type and the control mode; This solution builds a local command template library and combines it with matching rules to enable switching between online and offline template parsing modes. This allows the vehicle terminal to still parse and execute commonly used control commands of preset types even when the network is unavailable or of poor quality. Addressing the control blind spots caused by inconsistent third-party application APIs, this solution employs multimodal application control technology. By integrating a four-layer control system—URI Scheme, Broadcast, MQTT messages, and screen click simulation using accessibility services—it achieves full coverage of application functions. Furthermore, by integrating URI Scheme, broadcast, business messages, and simulated interactive operations based on system auxiliary functions, a multimodal application control mechanism is formed to improve adaptability to different types of vehicle infotainment application control interfaces.

[0009] To address the poor reliability of a single communication channel in complex environments, this solution designs a dual-channel adaptive communication architecture. Through redundant backups and health monitoring mechanisms for MQTT and WebSocket, a high command delivery rate is ensured. Compared to US7398209B2, this invention overcomes cloud dependency by using local template matching features, achieving a command recognition rate of over 90% in offline mode. Compared to US8214100B2, this invention overcomes communication interruption by using dual-channel adaptive switching features, increasing the command delivery rate to over 99.5%.

[0010] Preferably, in step S20, the multi-level parsing includes: using a first-level regular expression to identify the target application identifier, using a second-level regular expression to identify the instruction type, and using a third-level regular expression to extract the control parameters from the natural language instruction; using a third-level matching rule and a named entity recognition model to extract the control parameters from the natural language operation instruction, and automatically completing the missing business parameters in conjunction with the vehicle's current location and operation period; The named entity recognition model is a BiLSTM-CRF model, a BERT-CRF model, or a lightweight Transformer-CRF model. The training dataset includes mining transportation task corpus, logistics dispatch corpus, vehicle check-in corpus, navigation destination corpus, and industrial vehicle operation corpus. The rule dictionary includes an application identifier dictionary, a work action dictionary, a location dictionary, a task status dictionary, and a control text dictionary.

[0011] Preferably, in S30, the at least two communication channels with different protocol characteristics include a long-connection channel based on the MQTT protocol and a real-time duplex channel based on the WebSocket protocol; high-priority job instructions are transmitted concurrently through dual channels, while ordinary instructions are dynamically switched through a single channel.

[0012] Preferably, in step S30, the dynamic selection of the target channel includes: real-time monitoring of the latency, packet loss rate, and heartbeat response status of each communication channel; automatically switching to a backup channel and synchronizing status when the latency or packet loss rate of the current main channel exceeds a preset threshold; and multi-index weighted adaptation to the industrial vehicle status; the dynamic selection of one or more target communication channels includes: Real-time monitoring of latency, packet loss rate, and heartbeat response status of each primary and backup communication channel; and calculation of channel health score based on latency score and packet loss score. When the structured service control instruction is a high-priority operation instruction, it is transmitted concurrently through at least two communication channels; When the structured service control command is a normal command, the system dynamically switches between the main communication channel and the backup communication channel based on the channel health score, and performs status synchronization during the switching process. Wherein, the channel health score H = 0.6 × Ls + 0.4 × Ps, Ls = max[0, 1 - L / 500], Ps = max[0, 1 - P / 10]; Where H is the channel health score, Ls is the latency score, and Ps is the packet loss score. Both Ls and Ps are normalized dimensionless scores with values ​​ranging from 0 to 1. The larger the values ​​of Ls and Ps, the healthier the latency or packet loss status of the corresponding channel. L is the average latency of the most recent N heartbeats or business messages, in milliseconds. P is the packet loss rate of the most recent N heartbeats or business messages, in percentage. 500ms is the latency threshold for distinguishing between available links and congested links in a mining weak network test environment, and 10% is the packet loss rate threshold for distinguishing between available links and unstable links in a mining weak network test environment.

[0013] Preferably, in S40, the selection priority of the multiple preset control modes is as follows: URI Scheme control mode is higher than broadcast control mode, broadcast control mode is higher than business message control mode, and business message control mode is higher than simulated interactive operation based on OCR screen recognition, dynamic coordinate scaling and system auxiliary functions.

[0014] Preferably, before the multi-level parsing, a data cleaning step is also included: standardizing the original instruction text, including removing preset non-feature characters, filtering interjections, and unifying the text format; Preferably, after the multi-level parsing, a security verification step is also included: performing a compliance check on the call permission based on the target application identifier, instruction type, vehicle identity, and user permissions, and limiting the frequency of instruction calls by the same user, the same vehicle, or the same in-vehicle terminal based on a preset time window.

[0015] Preferably, the system also includes a data compensation step: in the offline mode, structured business control instructions that cannot be executed or reported in real time are added to a local execution queue, and conflict resolution is performed based on the instruction timestamp and the business priority queue; when the network is detected to be restored and the system switches to the online mode, the structured business control instructions in the local execution queue are synchronized to the server or business system. Preferably, it also includes business collaboration steps: the application operation status of the vehicle terminal is synchronized to the backend business management system in real time through specific protocol messages to trigger corresponding business check-in, task push operation, task confirmation, vehicle queuing or navigation linkage operation.

[0016] This invention provides an in-vehicle infotainment system based on screen recognition and command reconstruction, comprising: The mode switching module is used to automatically switch between online mode based on cloud parsing and offline mode based on local command templates based on network connection status monitoring results. The instruction parsing module is used to perform multi-level parsing of received natural language instructions based on preset matching rules in order to identify the target application identifier, instruction type and control parameters; The communication management module is used to maintain at least two communication channels with different protocol characteristics, and dynamically select the target channel to transmit the parsed instructions based on the real-time monitored channel health status. The execution control module is used to select the target control mode from a variety of preset control modes to execute the instruction based on the target application's support for the control interface. The variety of preset control modes includes simulated interactive operations based on system auxiliary functions as the underlying control. The system operates on an in-vehicle terminal, which includes a processor, microphone, GPS positioning module, in-vehicle communication module, and display screen. The system includes: The mode switching module is used to automatically switch between online mode based on cloud parsing and offline mode based on local command templates, based on the network connection status monitoring results obtained by the vehicle communication module. The instruction parsing and reconstruction module is used to perform multi-level business parsing and instruction reconstruction on the received natural language operation instructions based on preset matching rules, the current location of the vehicle, the operation period, and the industrial operation task context, so as to identify the target application identifier, instruction type, control parameters and business priority, and generate structured business control instructions. The communication management module is used to maintain at least two primary and backup communication channels with different protocol characteristics, and dynamically select one or more target communication channels to transmit the structured service control instructions based on the real-time monitored channel health status and the service priority. The execution control module is used to determine the control mode based on the parsed instruction type, and select the target control method from multiple preset control modes to execute the structured business control instruction based on the target application's support for the control interface. The multiple preset control modes include simulated interactive operations based on OCR screen recognition, dynamic coordinate scaling and system auxiliary functions as the underlying control. Technical problem solved by the present invention: (1) Solve the problems of high learning cost and poor scalability of fixed operation instructions for vehicles. Adopt natural language operation instruction parsing and business reconstruction technology, through multi-level matching rules, named entity recognition model and industrial operation rule dictionary, drivers can issue operation instructions such as "I want to clock in", "view the second task" and "navigate to loading point" in natural language, without having to memorize fixed formats. At the same time, the system can automatically complete missing business parameters by combining the vehicle's current location and operation time period. When adding new functions, it can be expanded by configuring local instruction templates, rule dictionary and business mapping relationship.

[0017] (2) Solve the problem of poor reliability of a single communication channel in a weak network environment of mines / logistics. At least two communication channels with different protocol characteristics are used. Through channel health monitoring, service priority scheduling and automatic switching mechanism, dual-channel concurrent transmission is used for high-priority operation instructions, and single-channel dynamic switching is used for ordinary instructions, thereby reducing the risk of operation instructions being lost due to weak network, network outage or single point of failure.

[0018] (3) Solving the problems of inconsistent application APIs, inconsistent interfaces of non-standard industrial business APPs, and limited functionality. This technical solution adopts multimodal application control technology, comprehensively utilizing URI Scheme, Broadcast, business messages, OCR screen recognition, and system auxiliary function simulation interaction to build a unified control framework. For non-standard mining APPs and freight APPs without API support or with closed interfaces, OCR recognition controls, dynamic coordinate scaling, and simulated click control are used to ensure the stability and integrity of key business functions.

[0019] (4) Addressing the issues of cloud-based processing relying on the network, offline unavailability, and security and privacy risks in industrial operations; This technical solution adopts an online / offline dual-mode instruction processing mechanism. When the network is normal, cloud-based enhanced understanding is used first, and when the network is weak or disconnected, it automatically switches to local instruction template matching and local business rule reconstruction, ensuring that mining / logistics industrial vehicles can still execute common operation instructions in environments without a public network or with a weak network. At the same time, sensitive business operations can be completed locally on the vehicle terminal, reducing the uploading of operation privacy data.

[0020] Compared with the prior art, the beneficial effects of the technical solution of the present invention include: (1) Improved accuracy of natural language understanding; Through multi-level regular expression matching and semantic analysis, the accuracy of natural language instruction parsing reaches over 95%, which is about 40% higher than that of fixed instruction word scheme. Supports complex business instructions such as "view the second task" and "I want to check in"; Through multi-level matching rules and semantic analysis, the matching accuracy between natural language job instructions and target applications, instruction types and control parameters can be improved; (2) Communication reliability is significantly improved; through at least two communication channels with different protocol characteristics and a channel health status monitoring mechanism, the system can switch to the backup channel when the main channel is abnormal, or use multi-channel concurrent transmission for high-priority instructions, thereby improving the reliability of instruction transmission; the dual-channel adaptive communication architecture increases the instruction delivery rate from 92% of the single channel to 99.5%, which is particularly significant in network fluctuation scenarios; ensuring the reliable delivery of key business instructions such as freight task push and check-in instructions; (3) Improved application control coverage; the multimodal application control framework supports more than 95% of vehicle-mounted application control functions, which is about 30% higher than the single API solution; through URI Scheme, broadcast, business messages and simulated interactive operations based on screen recognition and system auxiliary functions, the system's compatibility with different vehicle-mounted application control methods can be improved; especially for non-standard mining APP and freight APP, deep function control (check-in, task viewing, topic switching, etc.) is realized through MQTT messages; for non-standard business applications such as mining transportation application and freight dispatching application, control commands can be sent through preset business message topics and message formats to trigger business operations such as check-in, task viewing, and topic switching; (4) Deep interconnection between vehicle-mounted systems and business systems; Through the MCP server, the system is connected to the mine management system and the freight dispatching system, realizing: voice check-in and attendance system linkage; freight task push and vehicle-mounted navigation linkage; real-time synchronization of task status; closed-loop management of business data; Through the server, the system communicates with the mine management system or the freight dispatching system, enabling the voice check-in, task reception, task confirmation, navigation start and task status reporting operations of the vehicle terminal to be processed in coordination with the corresponding business system; The vehicle terminal establishes a data interaction relationship with the back-end business management system to realize the synchronization of application operation status, task status and business events; (5) Offline availability guarantee; The offline / online dual-mode processing mechanism ensures that common functions (music control, volume adjustment, etc.) can still be used normally when the network is unavailable, and the continuity of user experience is guaranteed; The online / offline dual-mode processing mechanism is used to call the local instruction template to parse the preset common instructions when the network is unavailable or the network quality is lower than the preset conditions, so as to maintain the availability of some key vehicle control functions. (6) Enhanced security; AES-256 encryption, HMAC-SHA256 signature, timestamp verification, Nonce anti-replay and other multiple security mechanisms significantly improve the security of instruction transmission and ensure the security of business data; during instruction transmission, one or more security verification methods such as encryption, message signature, timestamp verification and random number verification can be used to reduce the risk of instruction being tampered with, replayed or illegally invoked. (7) Enhanced scalability; the templated instruction parsing and plug-in application control framework shorten the development cycle of new functions from 2 weeks to 2 days and reduce maintenance costs by 70%; for new business applications that support business message control, access can be achieved by configuring the corresponding message topic, message format and control parameter mapping relationship; new business APPs only need to configure MQTT topic and message format; through templated instruction parsing rules and configurable application control mapping relationship, when adding new applications or new instructions, adaptation can be completed by configuring instruction templates, business parameters and control methods, thereby improving the convenience of system expansion.

[0021] This invention achieves coordinated control of natural language operation instructions from parsing, transmission, to execution through the collaborative processing of the semantic parsing layer, communication transmission layer, and execution control layer. Focusing on the reliable transmission and deterministic execution requirements of critical vehicle-mounted operation instructions in weak network environments, a three-layer linkage architecture is formed: At the semantic layer, cloud-enhanced parsing and local instruction template parsing are automatically switched, and business instructions are reconstructed based on the vehicle's current location and operation period; at the transmission layer, dual-channel concurrent transmission or single-channel dynamic switching is selected based on business priority and channel health status; at the execution layer, the control mode is determined based on the parsed instruction type, and in the absence of a unified API, fallback execution is achieved through OCR screen recognition, dynamic coordinate scaling, and simulated interaction using system auxiliary functions. The three-layer mechanism described above—the semantic parsing layer for generating structured service control commands, the communication transmission layer for selecting transmission channels based on channel health status and service priority, and the execution control layer for selecting control modes based on the target application interface support—works together to solve the problems of weak network conditions, lack of unified APIs, and high-reliability delivery of critical operation commands. Through this layered collaboration, the probability of command transmission failure in weak network environments can be reduced, and the control adaptation capability for applications without unified control interfaces can be improved. This has significant implications and remarkable effects.

[0022] The present invention provides a vehicle infotainment application method and system based on screen recognition and instruction reconstruction. Other advantages, objectives and features of the present invention will be partly apparent from the following description, and partly understood by those skilled in the art through research and practice of the present invention. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a diagram illustrating an embodiment of the overall architecture of a vehicle-mounted application system based on screen recognition and instruction reconstruction, as described in this invention.

[0024] Figure 2 This is a diagram illustrating an embodiment of the multi-level instruction parsing process of a vehicle-mounted application system based on screen recognition and instruction reconstruction, as described in this invention.

[0025] Figure 3 This is a diagram of an embodiment of the dual-channel adaptive switching logic of a vehicle-mounted system based on screen recognition and instruction reconstruction as described in this invention.

[0026] Figure 4 This is a diagram illustrating an embodiment of the multimodal application control flow execution logic of a vehicle-mounted application system based on screen recognition and instruction reconstruction, as described in this invention.

[0027] Figure 5 This is a diagram illustrating an embodiment of the offline / online dual-mode processing of an in-vehicle infotainment system based on screen recognition and command reconstruction as described in this invention.

[0028] Figure 6 This is a timing diagram of multi-device interaction in a vehicle-mounted application system based on screen recognition and command reconstruction, as described in this invention.

[0029] Figure 7 This is a diagram of an embodiment of the screen recognition sub-process of a vehicle-mounted application system based on screen recognition and instruction reconstruction as described in this invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, so that those skilled in the art can implement it based on the specification; as shown in the figures, the present invention provides a vehicle-mounted system application method and system based on screen recognition and command reconstruction, including: like Figure 1 As shown, the system architecture is divided into five core layers from top to bottom: application layer, control layer, communication layer, execution layer, and the bottom external system; these five layers enable highly reliable execution of critical operational commands from the vehicle's infotainment system. Application Layer: Primarily responsible for direct interaction between the vehicle's infotainment system and the driver, status display, and command input. It comprises three modules: Voice interaction interface (task status display): used to intuitively display the execution status of the current industrial operation task to the vehicle driver; Task management interface (user command input): Receives natural language control commands input by the vehicle driver (such as "I want to clock in" or "View the second task"); Natural Language Processing Module (Command Conversion): Converts spoken natural language speech commands into text signals that can be logically parsed; Control Layer: This is the core control hub of the vehicle's infotainment system, responsible for command deconstruction and reconstruction, as well as business continuity management in offline states. It comprises three modules: Command Parser: Based on preset matching rules, it performs multi-level parsing of input text commands to accurately identify the target application identifier, command type, and specific control parameters; CarApp Controller (Application Scheduling): Distributes and schedules the corresponding processor or execution mode based on the instruction type and the target application's support for the control interface; Foffline Processor: Automatically activates offline mode when the network is unavailable, matches local instruction templates to execute local functions, and adds control instructions that cannot be executed immediately to the local execution queue to resolve conflicts. Communication Layer: Responsible for highly reliable, low-latency data interaction between the edge and the cloud, employing protocol redundancy design to withstand harsh industrial environments; includes two primary and backup communication channels. Client: Mqut Control Client (MQTT protocol client): Maintains a long-lived TCP connection channel (standard port 1883) based on the MQTT protocol, supports QoS 1, and is used for regular command issuance and status reporting to ensure that messages are delivered at least once.

[0031] WebSocket Client Adapter: Maintains a real-time full-duplex channel (custom port 8081) based on the WebSocket protocol for transmitting urgent commands requiring extremely high real-time performance and high-frequency, fine-grained screen coordinate data.

[0032] Execution layer: Responsible for translating high-level logic instructions into actual control operations for various applications within the vehicle's infotainment system. It includes two complementary execution controllers: Amap API Navigator: For applications with publicly available standard interfaces (such as the Tudmap API), it prioritizes native and efficient control by constructing URI Schemes or sending system broadcasts.

[0033] Click Accessibility Service: As the underlying control fallback, it is designed for non-standard vertical industrial apps without API support. It utilizes OCR screen recognition controls and dynamic coordinate scaling to simulate human gesture clicks to execute business loops through the system's accessibility service.

[0034] External System: This belongs to the backend and cloud support system and works closely with the A11 application on the vehicle's infotainment system via a high-speed communication channel. It includes three components: MCP Server (Remote Control): The cloud-based control server serves as the "brain" of the entire system. It is deployed using a high-performance server cluster and is responsible for enhanced semantic understanding and structured business instruction reconstruction in online mode.

[0035] Network Monitor: Real-time monitoring of latency, packet loss rate, and heartbeat response status of each communication channel, calculating channel health scores, and providing underlying basis for mode switching and adaptive channel scheduling.

[0036] Business systems (mining / freight): The backend mining management system, freight dispatching system or attendance management system receives the synchronized operation status from the vehicle terminal to realize closed-loop data management of industrial operations.

[0037] The core interaction logic and data flow within the architecture; The system's deep control logic presents a closed-loop process of "multi-layer linkage": Application layer: Receives natural language (determines network status: sends to cloud MCP when online, retains data locally when in weak network conditions); Control layer: Command Parser / Foffline refactors instruction objects (determines business priority: high priority dual-channel concurrency, normal single-channel switching); Communication layer: MQTT / WebSocket client adaptive routing transmission (judging interface support: native interfaces are given priority, and non-standard APPs are provided as a backup for barrier-free operation); Execution layer: API Navigator / Click Accessibility drives the APP (synchronizes application status, triggers business check-in, task push, vehicle queuing); External system: Closed loop of mining / freight business system.

[0038] The internal logic and data flow of the architecture: 1. Routing and Restructuring Logic of the Semantic Layer (Application Layer - Control Layer) When a driver inputs complex operational instructions at the application layer via interface or voice, such as "Check in at loading point number one, then check the second task and navigate there," the command parser at the control layer will refactor the business logic based on the network status feedback from the network monitor in the external system. In online mode: Sensitive data is sent to the cloud-based MCP Server via the communication layer for enhanced semantic parsing.

[0039] In weak network / disconnected network conditions: The system automatically switches to offline mode, and the `Foffline Processor` matches local instruction templates and automatically completes missing business parameters by combining the vehicle's current location, operation time period and other contexts, and decomposes composite instructions into standardized atomic instruction chains with dependencies.

[0040] 2. Adaptive distribution logic of the transport layer (control layer — communication layer) The reconstructed structured business control instruction objects are sent to the communication layer. The communication layer performs adaptive scheduling based on the instruction's "business priority" and the channel's "health score." High-priority operation instructions (such as loading / unloading confirmation, safe evacuation): The system mobilizes `Mqut Control Client` and `Websocket Client Adapter` for dual-channel concurrent transmission to ensure absolute delivery rate in the weak network environment of the mining area.

[0041] Ordinary commands (such as music adjustment): The system dynamically switches between primary and backup channels based on the health score (weighted by latency and packet loss rate). If congestion occurs, the system adopts the "connect first, disconnect later" strategy to resend the command in the buffer, which is completely transparent to the driver.

[0042] 3. Priority degradation at the execution layer and external closed-loop logic (communication layer—execution layer—external system) After the command is successfully delivered to the vehicle's infotainment system, the CarApp Controller in the control layer controls the vehicle's applications according to a four-layer control system that prioritizes native interfaces and uses analog operations as a backup. If the target application supports public interfaces, the `Amap Api Navigator` will be driven first to make fast and efficient native calls via URIScheme or system broadcast.

[0043] If the target application is a non-standard industrial app that is completely closed and has no public API, then the `ClickAccessibility Service` module is downgraded and enabled. It uses screenshot OCR to identify specific business controls (such as the "Confirm Loading" button), scales the coordinates proportionally, and simulates human gestures to perform precise clicks, eliminating blind spots in control.

[0044] Finally, after the execution layer completes its operation, it synchronizes the latest application operation status of the vehicle terminal to the backend business system (mining / freight) in real time through specific protocol messages (such as adaptive messages), automatically triggering corresponding business check-in, task push confirmation, queue linkage and other operations, perfectly achieving the collaboration and closed loop of the enterprise digital management system.

[0045] This invention provides a method and system for in-vehicle infotainment systems based on screen recognition and command reconstruction. The system mainly consists of an MCP control server, communication service components, a backend industrial business system, and the vehicle-mounted A11 application. The communication service components include an MQTT Broker and a WebSocket Server. The MCP control server is used for enhanced semantic understanding and business instruction reconstruction in online mode, employing a server cluster deployment with a core CPU running the instruction reconstruction engine. The vehicle-mounted A11 application is installed on the vehicle's in-vehicle mobile operating system. The in-vehicle terminal includes a processor, microphone, GPS positioning module, in-vehicle 4G / 5G communication module, display screen, and storage. The MQTT Broker serves as an asynchronous instruction transmission component, and the WebSocket Server handles real-time bidirectional interactive data. The vehicle-mounted A11 application maintains at least two primary and backup communication channels with different protocol characteristics simultaneously through the in-vehicle 4G / 5G communication module, and selects single-channel switching or multi-channel concurrent transmission based on channel health status and service priority.

[0046] The overall architecture, components, model parameters, and embodiments of the system of this invention; The system's MCP control server cluster is deployed with Intel Xeon Silver 4210R-class CPUs (10 cores, 20 threads, 2.4GHz, capable of supporting tens of thousands of regular expression parsing requests per second), running an instruction refactoring engine based on Python 3.8 or later. The vehicle-mounted A11 application is installed on the in-vehicle Android system's MQTT Broker, employing EMQX Enterprise Edition v4.4, responsible for asynchronous instruction push. The WebSocket Server is built on Node.js's Socket.io framework to handle bidirectional interactive data requiring extremely high real-time performance. The MCP control server connects to the MQTT Broker and WebSocket Server via a 10 Gigabit Ethernet interface, ensuring internal data exchange latency is less than 1ms. The vehicle-mounted A11 application simultaneously establishes a long TCP connection (standard port 1883) with the MQTT Broker and a full-duplex connection (custom port 8081) with the WebSocket Server via the in-vehicle 4G / 5G communication module.

[0047] Figure 2 This diagram illustrates an embodiment of the multi-level instruction parsing process in this invention; this process is the core logic of the MCP control server, and its physical essence is to transform disordered natural language signals into ordered structured control vectors. Figure 2As shown, the multi-level instruction parsing process in this invention is the core logic of the MCP control server or vehicle terminal processor. Its physical essence is to transform disordered natural language operation instructions into ordered structured business control instructions. The parsing is divided into three layers: The first layer is application identifier recognition, which uses the AC automaton algorithm to quickly match identifiers such as "A11", "driver assistant", "freight assistant", and "dispatch system" in the industrial vehicle application keyword library; the second layer is intent classification, which uses regular expression capture groups and industrial operation rule dictionary to match core verbs such as "navigation", "check-in", "confirm loading", "view task", and "open" to determine the operation instruction type; the third layer is parameter extraction, which uses named entity recognition (NER) technology to extract specific locations, list indexes, operation time periods, or task parameters such as "loading point", "unloading area", "second task", "early shift", and "vehicle number". The NER model can employ a BiLSTM-CRF model, a BERT-CRF model, or a lightweight Transformer-CRF model. The training dataset includes corpora of mining transportation tasks, logistics dispatching, vehicle check-in, navigation destinations, and industrial vehicle operation data. The rule dictionary includes application identifiers, work actions, locations, task status, and control text. Through this multi-level parsing, the system can automatically complete a fuzzy command like "Open the driver assistant and check in" by combining the vehicle's current location and work period, providing data support for subsequent automated execution. This includes the target app package name, the work action "check in," the check-in type, the vehicle location, the work shift, and the target control feature "check in."

[0048] In local offline mode, the vehicle terminal parses the command according to the offline command template mapping table, as shown in the following example: Table 1: Offline Command Template Mapping Table The following is an example of a regular expression matching rule table: Table 2: Regular Expression Matching Rules Table Figure 3 This diagram illustrates an embodiment of the dual-channel adaptive switching logic in this invention. The dual-channel adaptive switching logic includes: channel health monitoring, service priority determination, and target channel selection. The system sends heartbeat packets every 20-60 seconds. In high-temperature summer environments, frequent heartbeats can increase the power consumption of the communication module and cause overheating and frequency reduction. The heartbeat interval can be dynamically adjusted between 20-60 seconds based on battery temperature. The system monitors the average latency L, packet loss rate P, and heartbeat response status of each communication channel in real time, and selects channels based on the channel health score H = 0.6 × Ls + 0.4 × Ps. Here, H is the channel health score; Ls is the latency score; and Ps is the packet loss score. Both Ls and Ps are normalized dimensionless scores, ranging from 0 to 1; a higher value indicates a healthier latency or packet loss status for the corresponding channel. Ls=max[0,1-L / 500], Ps=max[0,1-P / 10%], where L is the average delay of the most recent N heartbeats or business messages in milliseconds, and P is the packet loss rate of the most recent N heartbeats or business messages in percentage; 500ms is the delay threshold for distinguishing between available links and congested links in a mine weak network test environment, and 10% is the packet loss rate threshold for distinguishing between available links and unstable links in a mine weak network test environment; for example, when the average delay of the most recent N heartbeat messages of a certain communication channel is L=250ms and the packet loss rate is P=2%, the delay score Ls=max[0,1-250 / 500]=0.5, the packet loss score Ps=max[0,1-2 / 10]=0.8, and the channel health score H=0.6×0.5+0.4×0.8=0.62. If the health score of the other communication channel is 0.81, then for ordinary instructions, the communication channel with a health score of 0.81 will be selected first; for high-priority job instructions, they can be transmitted concurrently through two communication channels. When the system detects three consecutive heartbeat timeouts on the current main channel, an average latency exceeding 500ms, or a packet loss rate exceeding 10%, it determines that the current link is congested or disconnected. For high-priority work instructions, the system transmits them concurrently through at least two communication channels. For ordinary instructions, the system automatically switches the active channel to a backup channel with a higher health score. The switching process adopts a "connect first, then disconnect" strategy to ensure that undelivered instructions are resent in the buffer of the new channel, solving the pain point of lost work instructions in mining / logistics industrial environments. When the system detects three consecutive heartbeat timeouts on the MQTT channel (i.e., no response for 90 seconds) and an average latency exceeding 500ms, it determines that the current link is congested or disconnected and automatically switches the active channel to the WebSocket Server. The switching process is completely transparent to the user, ensuring that undelivered instructions are resent in the buffer of the new channel, solving the pain point of lost instructions in complex industrial environments. This logic achieves high availability of the communication link by monitoring the latency, packet loss rate, and heartbeat response status of each communication channel in real time, ensuring the delivery rate of control instructions from the vehicle terminal in weak network environments.

[0049] Figure 4This diagram illustrates an embodiment of the multimodal application control flow execution logic in this invention. This module resides within the A11 application on the vehicle's infotainment system and employs a tiered degradation strategy to achieve maximum compatibility. First, it detects the target application's support for the URI Scheme. This is similar to sending the application a URL with parameters, such as "A11auto: / / route / plan?dest=lat,lon", which is the most efficient and stable method. If the target application has not registered the corresponding Scheme, it performs a degradation-based detection and uses system broadcast for communication. For completely closed third-party interfaces that do not provide any interfaces, the Accessibility Service is activated. The Accessibility Service defines the click path using GestureDescription.StrokeDescription, simulating the pressing and releasing of coordinates (x, y). The coordinate calculation formula is: X_real = X_cmd (Width_screen / 1920), Y_real = Y_cmd (Height_screen / 720) (dynamically scaled and adapted according to the actual screen resolution, such as 1280x720 or 2K screen, to ensure the click position does not deviate from the button center). This multimodal solution, through its redundant design, ensures that regardless of how closed off third-party applications may be, the system can achieve a closed business loop by simulating manual operation.

[0050] In this invention, the multimodal application control flow execution logic is located within the A11 application on the vehicle-mounted system, employing a tiered degradation strategy to achieve maximum compatibility. First, the target control modality is determined based on the parsed instruction type, and the target application's support for the URI Scheme is detected, for example, "A11auto: / / route / plan?dest=lat,lon". If the target application has not registered the corresponding Scheme, degradation detection is performed, and system broadcast is used for communication. If the target application is a non-standard mining or freight app that supports business messages, check-in, task confirmation, task viewing, or status synchronization are completed through business messages. For completely closed third-party interfaces that do not provide any interfaces, OCR screen recognition and AccessibilityService are activated.

[0051] An example of a multimodal control priority parameter table is as follows: Table 3: Multimodal Control Priority Parameter Table like Figure 5As shown, this invention employs an offline / online dual-mode processing mechanism. When the network is normal, the vehicle terminal prioritizes cloud-based enhanced parsing; when the network is weak or down, it automatically switches to local instruction template matching and local business rule reconstruction. In offline mode, structured business control instructions that cannot be executed or reported immediately are added to a local execution queue, and conflicts are resolved based on instruction timestamps and business priority queues. Higher-priority instructions are executed before ordinary instructions, and instructions of the same priority with earlier timestamps are executed first. If mutually exclusive instructions exist, valid instructions are retained according to business rules, or the driver is prompted for confirmation. When network recovery is detected and the system switches to online mode, the structured business control instructions in the local execution queue are synchronized to the server or business system.

[0052] like Figure 6 As shown, this invention synchronizes the application operation status of the vehicle terminal to the mine management system, logistics dispatch system, or attendance management system in real time through business collaboration steps, thereby triggering corresponding business check-in, task push, task confirmation, vehicle queuing, or navigation linkage operations. For example, after successful voice check-in, it is synchronized to the attendance management system; after confirming loading, it is synchronized to the mine management system; after receiving the task, it is synchronized to the logistics dispatch system; when navigating to the loading point, the synchronized task status is "heading to loading point".

[0053] In terms of security verification, after multi-level parsing of business logic and before instruction execution, the system performs a compliance check on call permissions based on the target application identifier, instruction type, vehicle identity, and user permissions. It also limits the frequency of instruction calls from the same user, vehicle, or in-vehicle terminal within a preset time window. For example, if the same vehicle initiates more than a preset number of "confirm loading" instructions within 60 seconds, the system determines this as an abnormal frequency and refuses execution or requires secondary confirmation.

[0054] In a weak network testing environment in a mine, the accuracy rate of natural language operation instruction parsing can reach over 95%, a significant improvement compared to fixed operation instruction word solutions. It supports industrial business instructions such as "View the second task," "Check-in," and "Navigate to loading point." In the same environment, the dual-channel adaptive communication architecture can increase the delivery rate of critical operation instructions to over 99.5%, offering higher reliability under network fluctuations compared to single-channel communication methods. This ensures reliable delivery of critical business instructions such as freight task pushes, operation check-ins, and loading / unloading confirmations. In a vehicle-mounted system testing environment, the multimodal application control framework enhances the controllability of non-standard mining apps, freight apps, and third-party closed interfaces. For functions without a unified API, fallback control can be achieved through OCR screen recognition, dynamic coordinate scaling, and system auxiliary functions simulating interaction.

[0055] like Figure 7As shown, in the OCR screen recognition and dynamic coordinate scaling sub-process, the system first performs OCR recognition on the current interface screenshot to obtain the candidate control text, control area coordinates, and confidence level. It then matches and verifies the target control semantics in the structured business control instructions with the OCR recognition results. Once the target control is determined, dynamic coordinate scaling is performed based on the current screen resolution and template resolution. The scaling formula is x'=x×Wc / Wt, y'=y×Hc / Ht, where x and y are the coordinates of the target control's center point at the template resolution, x' and y' are the coordinates of the target control's center point at the current screen resolution, Wt and Ht are the width and height at the template resolution, and Wc and Hc are the width and height at the current screen resolution. Subsequently, the accessibility service defines the click path through GestureDescription.StrokeDescription to simulate the pressing and releasing of the coordinate point (x', y').

[0056] Compared to US7398209B2, this invention, in mining / logistics industrial vehicle-mounted terminal scenarios, overcomes the shortcomings of relying solely on cloud parsing in weak or offline environments by using local instruction templates, an industrial operation rule dictionary, and a mechanism to complete the information based on the vehicle's current location and operation time. It can still complete the identification and reconstruction of commonly used operation instructions even offline. Compared to US8214100B2, this invention overcomes the defect of losing critical industrial operation instructions due to a single communication link interruption by using differentiated communication transmission based on business priority—namely, dual-channel concurrent transmission of high-priority operation instructions and dynamic switching of ordinary instructions via a single channel. This improves the reliability of critical operation instruction delivery in weak network testing environments in mines. In one embodiment, this invention provides a vehicle-mounted terminal control method. This method first obtains the network connection status in real time through the network monitoring module in the A11 application on the vehicle-mounted terminal, for example, by listening to the ConnectivityManager.CONNECTIVITY_ACTION broadcast of the A11 system to obtain the current RSSI (Received Signal Strength Indicator) and RSRP (Reference Signal Received Power) values. When the RSRP (Responsive RSRP) remains below the set RSRP threshold (which can be fine-tuned based on the sensitivity of different communication modules, such as Quectel EC20, ranging from -105dBm to -115dBm) for more than 3 seconds, the system automatically switches from the cloud-based online mode to the local command template-based offline mode. In offline mode, the A11 application on the vehicle's infotainment system calls the built-in SQLite database to match the preset local command template. Next, the system performs multi-level parsing on the received natural language commands, identifying the target application identifier (e.g., "com.example.app"), command type (e.g., "OPEN_AND_CLICK"), and control parameters (e.g., coordinates or button ID). Subsequently, the system maintains two communication channels with different protocol characteristics: MQTT and WebSocket, and dynamically selects the target channel to transmit the parsed command based on the real-time monitored channel health score (HealthScore = 0.6 connection rate + 0.4 reciprocal of latency). Finally, based on the target application's support for the control interface, the optimal mode is selected from various preset control modes such as URI Scheme, broadcast, and system auxiliary functions for execution. This method achieves stable control in complex network environments, solving the problem of uncontrollable vehicle-mounted systems in weak network scenarios such as tunnels or mining areas.

[0057] Part 1: Methodology and Technical Solutions a. Detailed description of the method and process This technical solution provides a vehicle infotainment intelligent control method based on screen recognition and command reconstruction, including the following steps: Step S1: Command Reception and Preprocessing Receiving a natural language control instruction text from a user terminal, and preprocessing the instruction text, comprising: Removing leading and trailing blank characters Converting to lowercase format (Chinese instructions remain unchanged) Removing modal particles (such as "ba", "ne", "a", etc.) Step S2: Natural language instruction parsing Parsing the preprocessed instruction text by using a multi-level regular expression matching algorithm to identify instruction types and extract parameters: S2.1: First-level matching - application identifier identification Detecting whether the instruction contains a specific application identifier (such as "A11", "B11 Map", "E11 Platform", etc.) If the application identifier is included, entering the special instruction parsing branch of the application S2.2: Second-level matching - instruction type identification Navigation-type instructions: matching patterns such as "navigate to (.+)", "from (.+) (navigate to|to|go to|head for)(.+)", etc. Music-type instructions: matching patterns such as "play (.+)", "pause", "next song", etc. Volume-type instructions: matching patterns such as "volume (increase|turn up|decrease|turn down)", "volume.(\d+)", etc. Application-type instructions: matching patterns such as "open (.+)", "close (.+)", etc. S2.3: Third-level matching - parameter extraction Extracting instruction parameters by using regular expression capture groups For navigation instructions, extracting the names of the starting point and end point positions For music instructions, extracting the song name For volume instructions, extracting the volume level value S2.4: Constructing a structured instruction object Creating a Command object, which includes attributes such as instruction type (CommandType enumeration), parameters, starting position, etc. Instruction types include: NAVIGATION, MUSIC_PLAY, MUSIC_SEARCH, VOLUME_UP, VOLUME_SET, APP_OPEN, etc.; S2.5: Command Reconstruction Instruction reconstruction is a process of converting a natural language voice instruction issued by a user into a standardized atomic instruction sequence executable by a vehicle-machine system after semantic parsing; the process comprises the following core links: (a) Complex instruction decomposition: breaking down a complex instruction containing multiple intentions into multiple independent atomic instructions. Sequence conjunction recognition: Recognizes conjunctions such as "first...then...", "then", "next", and "after". Parallel intent recognition: Recognizes parallel words such as "at the same time," "and," and "also." Conditional intent recognition: Recognizes conditional words such as "if...then..." and "when...then..." (b) Context parameter adjustment: Dynamically adjust command parameters based on the current vehicle system status, historical operation records, and environmental information. (c) Execution order arrangement: Determine the dependencies and execution order between atomic instructions. (d) Intermediate state management: Maintain the state machine during multi-step instruction execution to ensure that subsequent instructions are triggered only after the preceding instructions are completed; Example of instruction breakdown: User's original voice command: "Go to the gas station first, then go to the company." Decomposition process: Semantic parsing identifies the conjunction "first...then..." Break it down into two atomic instructions: Atomic instruction 1: {type: "NAVIGATION", target: "gas station", priority: 1} Atomic instruction 2: {type: "NAVIGATION", target: "Company", priority: 2} Execution process and intermediate state management: 1 IDLE → EXECUTING_STEP_1 Start navigation to "Gas Station" Command reception 2 EXECUTING_STEP_1 Listen to navigation status Continuous monitoring 3 EXECUTING_STEP_1 → STEP_1_COMPLETED Mark the first navigation segment as complete. Arrival at gas station / User confirmation 4 STEP_1_COMPLETED → EXECUTING_STEP_2 Start navigation to "Company" Precedence completion confirmation 5 EXECUTING_STEP_2 → COMPLETED Overall task completed Arrival at the company Example of context parameter adjustment: If the vehicle has sufficient fuel (obtained from the vehicle's OBD data), the system can prompt the user, "Fuel is sufficient. Do you want to skip the gas station and go directly to the company?" If the "company" address is not in the history, the system will complete the address by associating it with the user's frequently used location database. The navigation target "gas station" will be automatically associated with the nearest gas station POI or a specific target gas station POI. Specific target gas stations include: the gas station with the shortest travel time and the gas station with the least increase in distance. Step S3: Command Security Verification Perform security verification on the parsed structured instructions: S3.1: Permission Check Check if the current user has permission to execute this command. High-privilege commands (such as system settings modifications) require additional confirmation. S3.2: Frequency Limit Check Check if the instruction execution frequency exceeds the threshold. Preventing malicious high-frequency calls S3.3: Command Validity Verification Verify that the parameter format is valid. Check if the application package name exists in the whitelist. Step S4: Dual-channel communication transmission The verified structured instructions are sent to the vehicle's infotainment system via a dual-channel communication architecture: S4.1: Channel Selection Query the vehicle-mounted system channel status cache to obtain the currently active channels (MQTT or WebSocket). If the active channel is MQTT and the MQTT connection is normal, select the MQTT channel. If the active channel is WebSocket and the WebSocket connection is normal, select the WebSocket channel. If the active channel is unavailable, try the alternative channel. S4.2: Instruction Encapsulation Encapsulate structured instructions into JSON format messages. Message format: {"type": "command", "command": "command text", "timestamp": timestamp} S4.3: Message Sending MQTT Channel: Publish to the topic "car-control / {car_sn} / command" using the Paho MQTT client. WebSocket Channel: Sending Text Messages Using a WebSocket Connection S4.4: Send confirmation Waiting for confirmation of message delivery; If transmission fails, mark the channel as abnormal and trigger a channel switch.

[0058] Step S5: Receiving and parsing commands from the vehicle's infotainment system The vehicle-mounted system receives commands via a dual-channel client: S5.1: Channel Monitoring MQTT clients subscribe to the topic "car-control / {car_sn} / command" WebSocket client listens for text messages S5.2: Message Parsing Parse JSON format messages and extract instruction text Use CommandParser to re-parse the command text and generate a Command object.

[0059] Step S6: Determine offline / online mode Determine the current network status and select the processing mode: S6.1: Network Status Detection Detecting network availability using NetworkMonitor If network access is available, enter online mode (S7). If the network is unavailable, enter offline mode (S8).

[0060] Step S7: Online mode command execution Execute the command while the network is available: S7.1: Instruction Dispatch The command is dispatched to the corresponding processor based on the CommandType. Navigation commands → Navigation controller Music commands → Music controller Application Commands → Application Controller S7.2: Application Control Execution (Multimodal Control) Select the optimal control method based on the target application and instruction type: Method 1: URI Scheme (highest priority) Constructing an application-specific URI Scheme A11 Map Navigation: "A11auto: / / route / plan?sourceApplication=xiaozhi&dlat={lat}&dlon={lon}&dname={name}" B11 Map Navigation: "B11XXmap: / / map / navi?query={destination}" C11 Music Search: "C11music: / / C11.com / ui / search?key={song}" Scan the QR code on the E11 platform: "weixin: / / scanqrcode" Non-standard mining app (driver assistant): Launched via package name "com.example.aidrivermine" Non-standard freight APP (Freight Assistant): Launched via package name "com.driver.driver_ai" Launch the corresponding application via Intent Method 2: Broadcast method Send application-specific broadcast Intent A11 goes home: "com.autonavi.amapauto.AUTO_WIDGET_GO_HOME" A11 goes to the company: "com.autonavi.amapauto.AUTO_WIDGET_GO_COMPANY A11 search for gas stations: "com.autonavi.amapauto.AUTO_WIDGET_ADD_OIL" Method 3: MQTT messaging (business system integration) Deep functional control is achieved through communication with non-standard business apps via MQTT messages; Mine app check-in: Send an MQTT message to the topic "aidrivermine / startWork / {car_sn}", with the message content {"msg": "Check-in"}. Freight App Topic Switching: Send an MQTT message to the topic "driverai / changestyle / {car_sn}", with the message content {"type": "switch"}. Freight App Task Details: Send an MQTT message to the topic "driverai / opentaskdetail / {car_sn}", with the message content {"type": "open", "sortIndex": 2} Freight App Task: Send an MQTT message to the topic "driverai / opentaskdetail / {car_sn}", with the message content {"type": "pending"}. Process: First, open the target APP via the URI Scheme, wait 2-10 seconds, and then send an MQTT message to trigger the specific business function. In-depth details of the interaction between the MCP server interface protocol and business systems The MCP (Model Context Protocol) server uses the JSON-RPC 2.0 protocol format and communicates with the vehicle's infotainment system via both MQTT and WebSocket channels.

[0061] Request message format: { "jsonrpc": "2.0", "method": "car_control / execute", "params": { "device_id": "VEHICLE_001", "command": "navigation", "target": "Mining face A in mining area", "priority": "high", "timestamp": 1699000000000}, "id": "req-uuid-001" } Response message format: { "jsonrpc": "2.0", "result": { "status": "success", "execution_id": "exec-uuid-001", "message": "Navigation instructions executed" }, "id": "req-uuid-001"} Communication channel selection strategy: MQTT channel (MqttControlClient): Used for regular command issuance and status reporting, supporting QoS 1 to guarantee that messages are delivered at least once. WebSocket channels (WebSocketClientAdapter): Used for scenarios with high real-time requirements (such as emergency stop commands), providing full-duplex, low-latency communication. Business system data interaction process: The mining business system is interconnected: the mining scheduling system calls the MCP Server (Python) via HTTP / REST API. The MCP Server sends the data to the vehicle's MqttControlClient via MQTT Publish, which is then parsed by CommandParser and executed by CarAppController. Finally, the UI is operated through ClickAccessibilityService. A WebSocket channel is used as a backup.

[0062] Logistics business system linkage: The logistics TMS system notifies the MCP Server through Webhook callback. The MCP Server parses the waybill information to generate navigation instructions. After receiving the task push, the vehicle terminal automatically executes navigation to the pickup point and then to the delivery point.

[0063] Task push processing mechanism: When the business system pushes a task to the vehicle's infotainment system through the MCP server: Voice broadcast (TTS automatic broadcast): After receiving the task push message, the vehicle's infotainment system extracts the task summary information and calls the A11 TTS engine (TextToSpeech) to broadcast the message, such as "New task received: Please proceed to loading point B in the mining area, estimated distance 5.2 kilometers"; Pop-up notification: Add a system-level floating window (TYPE_APPLICATION_OVERLAY) via WindowManager to display task details and provide "Execute Now" / "Remind Later" / "Reject Task" action buttons; Automatic execution strategy: High-priority tasks (such as safe evacuation) are executed automatically with only voice notification; ordinary tasks are executed after confirmation in a pop-up window; low-priority tasks are added to the queue and prompted after the current task is completed.

[0064] Method 4: Media Key Method Use AudioManager.dispatchMediaKeyEvent to send media key events; It supports PLAY, PAUSE, NEXT, PREVIOUS and other buttons.

[0065] Method 5: Accessibility Service – Smart Control Based on Screen Recognition Obtain control tree information of the current interface of the target application in real time; Identify the location of the target control; Generate a control path based on the target control's location; Execute the corresponding gesture action; Dynamically adjust subsequent control paths based on interface feedback results; The A11 AccessibilityService framework implements real-time recognition and gesture simulation control of screen UI elements, as detailed below: (a) UI element tree traversal mechanism ClickAccessibilityService inherits from AccessibilityService. The following steps complete the traversal of the UI element tree: Call getRootInActiveWindow() to get the root AccessibilityNodeInfo node of the currently active window; Perform a depth-first traversal (DFS) on the root node, parsing the attribute information of the child nodes level by level: getText() — Gets the text content of a node; getContentDescription() — Gets the content description; getClassName() — Gets the control type (such as Button, EditText, TextView); getBoundsInScreen(Rect) — Gets the absolute coordinate boundaries of a node on the screen; isClickable() / isScrollable() — Determines the interactivity of a node; Match the attributes of the traversed nodes with the instruction target (text matching, control type matching) to locate the target UI element; (b) Coordinate dynamic calculation algorithm To address the issue of UI element coordinate offset under different vehicle infotainment screen resolutions, a proportional scaling algorithm based on reference coordinates is adopted: ActualX = BaseX × (ActualWidth / BaseWidth) ActualY = BaseY × (ActualHeight / BaseHeight) BaseWidth=1024 and BaseHeight=600 are the base resolution dimensions.

[0066] When AccessibilityNodeInfo can directly obtain the BoundsInScreen of the target node, the center point of the node's actual coordinates should be used first: CenterX = (bounds.left + bounds.right) / 2 CenterY = (bounds.top +bounds.bottom) / 2 (c) Gesture simulation mechanism Gesture simulation can be achieved using the AccessibilityService.dispatchGesture() method: 1. Create a Path object and set the starting point coordinates using moveTo(x, y); 2. Use GestureDescription.StrokeDescription(path, startTime, duration) to encapsulate the gesture parameters (click duration=100ms); 3. Assemble the complete gesture using GestureDescription.Builder().addStroke(strokeDescription).build(); 4. Call dispatchGesture(gestureDescription, callback, handler) to execute the gesture, and listen for the execution result through the callback; For swiping operations, Path additionally calls lineTo(endX, endY) to set the endpoint, and the duration is set to 300-500ms; (d) Multi-resolution adaptation table Baseline resolution 1024 600 1.000 1.000 Mining vehicle-mounted terminal (standard) High resolution 1280 720 1.250 1.200 Logistics vehicle terminal Full HD resolution 1920 1080 1.875 1.800 High-end passenger vehicle infotainment system low resolution 800 480 0.781 0.800 Economic vehicle-mounted equipment Suitable for functions without API support (such as map zooming, view switching, etc.).

[0067] S7.3: Execution Result Feedback Collect application execution results Send the results to the server via the active channel. Result format: {"type": "result", "command": "command", "success": true / false, "message": "result message"}.

[0068] Step S8: Offline mode command execution Execute commands when the network is unavailable: S8.1: Local Template Matching Load pre-built templates from the local command template library (command_templates.json) Use regular expressions to match user commands and templates. Template format: {"pattern": "regular expression", "action": "action", "parameters":{parameters}} S8.2: Offline command execution Once a match is successful, CarAppController will execute the corresponding action. It only supports locally executable commands (such as music control, volume adjustment, etc.). S8.3: Local POI Data Caching and Offline Navigation Cache all POIs (latitude, longitude, name, and category) visited by the user in the last 30 days. Cache the route points (load points, unload points, gas stations, rest areas) related to the current task. Cache data is stored in a local SQLite database and is periodically synchronized with the server. Offline navigation execution example (scenario: vehicle enters mining tunnel / underground area, network completely interrupted): User voice command: "Navigate to the loading point" Execution process: 1. CommandParser local parsing commands → {type: NAVIGATION, target: "mount point"}; 2. OfflineCommandProcessor checks network status → Offline mode; 3. Query local POI cache → Matched "Loading point A-3 in mining area" (most recently visited); - Longitude: 116.397428, Latitude: 39.90923; - Last accessed: 2 hours ago; 4. Use the G11 map offline navigation SDK → Use the downloaded offline map package; 5. ClickAccessibilityService simulates operation of the G11 map interface; 6. Navigation started successfully → Voice feedback: "Offline navigation has been planned for you to loading point A-3 in the mining area"; 7. Instruction execution records are stored in the synchronization queue; 8. After network recovery → Automatically report execution results to MCP Server; Offline mode experimental data: Frequently Used Routes (Visits in the last 7 days) 95% 98% 96% 1.2s Less frequently used routes (last 30 days) 92% 85% 82% 1.5s Uncached routes 90% 0% (Navigation failed) N / A 0.8s Synchronization after network recovery N / A N / A 100% 2.0s S8.4: Instruction Cache Instructions that cannot be executed offline are added to the pending queue. The network will automatically synchronize to the server after it is restored.

[0069] Step S9: Channel Status Management and Maintenance Continuously maintain the dual-channel communication status: S9.1: Heartbeat Maintenance MQTT channel: Sends heartbeat messages to "car-control / {car_sn} / heartbeat" every 20-60 seconds. WebSocket channel: Sends ping messages every 20 seconds S9.2: Access Control Health Check Regularly check the connection status of each channel. Detection channel latency and packet loss rate S9.3: Channel Switching Decision – Dual-Channel Adaptive Switching Strategy This invention employs a dual protection mechanism of "active switching + passive fault tolerance": Active switching implementation (based on packet loss rate threshold): Packet loss rate ≥5% Statistics every 10 seconds Trigger channel switching assessment Consecutive timeouts ≥3 times Real-time counting Switch to backup channel immediately RTT Delay ≥200ms Sample every 5 seconds Mark channel quality degradation Switching decision-making algorithms: if (packetLossRate >= 5% && consecutiveTimeout >= 2) {triggerChannelSwitch(); / / Actively switch } else if (packetLossRate >= 3%) { markChannelDegraded(); / / Mark downgraded, preparing for switching. } Passive fault tolerance (automatic switching upon connection loss): MQTT connection disconnection event triggered → Automatically switch to WebSocket channel within 200ms WebSocket connection disconnection event trigger → Automatic switching to MQTT channel within 200ms performance metrics: Active switching time Average 150ms <200ms Passive fault-tolerant switching time Average 180ms <200ms Switching success rate 99.2% >99% Message loss rate during handover 0.3% <1% Time required for switchback after channel recovery Average 500ms <1000ms NetworkMonitor monitoring implementation: Registers network status callbacks through ConnectivityManager; periodically sends heartbeat packets to check channel availability (MQTT: PINGREQ, WebSocket: Ping frames); maintains channel quality scores (0-100), comprehensively considering packet loss rate, latency, and bandwidth utilization; after MQTT recovery, decides whether to switch back to MQTT based on configuration; updates the channel status cache and notifies the server during switching.

[0070] S9.4: Channel status reporting; Regularly report the channel status to the server; Status format: {"activeChannel": "MQTT / WEBSOCKET", "mqttConnected": true / false, "wsConnected": true / false} b. Key steps; The key steps of this technical solution include: Key Step 1: Multi-level parsing of natural language instructions (S2); This step is central to solving the natural language understanding problem. Unlike existing technologies that use fixed instruction word matching, this technical solution employs a three-level regular expression matching architecture: The first level identifies the application identifier and enables application-level routing. The second level identifies instruction types and supports complex semantic patterns; The third-level parameter extraction supports multi-parameter capture; the key technical feature of this step is: Dynamic parameter extraction using regular expression capture groups Supports optional parameters (such as starting position). Supports synonym expansion (e.g., "increase" / "enlarge" / "bigger"). Key Step Two: Dual-Channel Adaptive Communication (S4, S9) This step is crucial for solving reliable communication problems in complex network environments. Unlike existing technologies that rely on a single channel, this solution maintains two channels simultaneously: MQTT and WebSocket. Channel priority is configurable (MQTT is prioritized by default). Real-time monitoring and caching of channel status Automatic fault detection and channel switching Channel status bidirectional synchronization The key technical feature of this step is: Both the server and client maintain channel state caches. Seamless channel switching (no commands lost) Supports channel health checks and automatic recovery. Key Step 3: Multimodal Application Control (S7.2) This step is central to solving the problem of unified control over third-party applications. Unlike the single API call of existing technologies, this technical solution constructs a four-layer control system: URI Scheme: Standard, Reliable, and Fully Functional Broadcast method: lightweight, fast, system-level support Media Key method: Universal and highly compatible Accessibility Service approach: a fallback, full functional coverage. The key technical features of this step are: Control methods: priority sorting and automatic degradation Coordinate-adaptive screen clicks (adapting to different resolutions) Control result feedback and failure retries Key Step 4: Offline / Online Dual-Mode Processing (S6-S8); This step is the core of solving the offline scene control problem; unlike the pure cloud processing of existing technologies, this technical solution implements a local instruction template library: pre-set commonly used instruction templates; local regular expression matching; pending instruction queue and network synchronization; c. Multiple methods, procedures, and solutions; This technical solution provides the following multiple methodologies and processes: Option 1: Cloud-first mode (default option); Process: User voice → Cloud NLP parsing → Dual-channel transmission → In-vehicle system execution; Features: High accuracy in natural language understanding; supports complex semantics and context; relies on network connectivity; applicable scenarios: urban areas with good network coverage.

[0071] Option 2: Local Priority Mode (Offline Solution); Process: User voice → Local command matching → Local execution Features: Does not rely on a network; fast response speed; limited command support; Applicable scenarios: Areas with weak network signals (underground parking lots, remote areas).

[0072] Option 3: Hybrid Mode (Combining Options) process: When network is available: Cloud-based NLP parsing → Dual-channel transmission → Vehicle-mounted system execution When the network is unavailable: Local command matching → Local execution → Synchronization after network recovery Features: Balancing accuracy and usability Automatic mode switching Best user experience Applicable scenarios: Suitable for all scenarios Option 3 effect: 1. When network access is available, use cloud-based NLP to ensure the accuracy of natural language understanding. 2. Use a local template when the network is unavailable to ensure basic functionality is available. 3. Automatic mode switching, no user intervention required. 4. After offline commands are restored, synchronize them to ensure data consistency. d. Multi-device interaction process This technical solution involves interaction between multiple devices, including user terminals, MCP servers, MQTT Brokers, WebSocket Servers, and in-vehicle infotainment systems. The interaction sequence is as follows: e. Formula Explanation This technical solution involves the following key algorithm formulas: (1) Instruction matching similarity formula Used for command template matching in offline mode: Similarity(T, I) = α·PatternMatch(T, I) + β·KeywordMatch(T, I) +γ·SemanticMatch(T, I) in: T stands for: Instruction Template I represents: Input command PatternMatch(T, I) represents the regular expression matching degree (0 or 1). KeywordMatch(T, I) represents the keyword matching degree (between 0 and 1). SemanticMatch(T, I) represents semantic similarity (calculated using word vectors). α, β, and γ represent weighting coefficients, with values ​​including α=0.6, β=0.3, and γ=0.1. (2) Channel health scoring formula Used for channel selection and switching decisions: HealthScore = w1·Connectivity + w2 · (1 / Latency) + w3 · (1-PacketLoss) + w4·Stability in: Connectivity represents the connection status (0 or 1). Latency stands for delay (milliseconds). PacketLoss represents the packet loss rate (between 0 and 1). Stability represents a stability score (based on the number of historical disconnections). w1, w2, w3, and w4 represent: weighting coefficients (3) Volume calculation formula Commands for volume settings: TargetVolume = (MaxVolume × Percentage) / 100 in: MaxVolume represents the system's maximum volume value (obtained via audioManager.getStreamMaxVolume). Percentage represents the user-specified volume percentage (0-100). TargetVolume represents the target volume value.

[0073] An example: A composite intent reconstruction example based on industry context input In this embodiment, the vehicle-mounted terminal A11 is installed on the vehicle-mounted terminal of the mining transport vehicle. The vehicle-mounted terminal includes a microphone, a GPS positioning module, a vehicle-mounted 4G / 5G communication module, a display screen, and local storage. The vehicle's current task is "loading ore from loading point one and transporting it to unloading point three." The vehicle is currently en route to loading point one, and the network status is weak but not completely disconnected.

[0074] The driver inputs the following natural language operation instructions via voice: "After arriving at loading point number one, check in first, then check the second task and navigate there." The input data received by the system includes: Voice-recognized text: "After arriving at loading point number one, check in first, then check the second task and navigate there." Current vehicle location: Approximately 1.2 kilometers from loading point number one; Current work phase: en route to the loading point; Current driver status: logged in and has check-in privileges; Current task list: Contains at least two tasks to be executed; Network status: MQTT channel is available, WebSocket channel has high latency; Target application status: Attendance APP can be controlled through business messages, Task APP does not provide a complete API, and Map APP supports URI Scheme.

[0075] The system first cleanses the natural language text, removing non-business-related expressions and identifying locations, actions, objects, and connections. Specifically, "after arriving at loading point number one" is identified as a location trigger condition, "first" and "then" are identified as sequential relationships, and "check in," "view the second task," and "navigate there" are identified as three business actions. Because "navigate there" depends on the destination information of the "second task," the system treats "view the second task" and "navigate there" as consecutive actions with data dependencies, rather than simply executing them in parallel.

[0076] The system output of the composite intent reconstruction result includes three atomic business instructions. The first atomic business instruction is "Check in upon arrival at loading point number one," targeting the attendance app. Its precondition is that the vehicle enters the electronic fence of loading point number one, and its completion condition is that the attendance system returns a successful check-in or a check-in record to be synchronized is generated locally. The second atomic business instruction is "View the second task," targeting the task app. Its precondition is that the first atomic business instruction is completed, and its completion condition is successfully entering the details page of the second task or retrieving the data for the second task. The third atomic business instruction is "Navigate to the destination of the second task," targeting the map app. Its precondition is that a valid destination exists in the second task, and its completion condition is that the map app enters navigation mode.

[0077] During execution, since the vehicle has not yet entered the electronic fence of loading point number one, the first atomic business instruction enters a waiting state. The system continuously monitors the vehicle's location. Once the vehicle enters the electronic fence of loading point number one, the system switches the first atomic business instruction to a ready state and sends a check-in request to the attendance system via MQTT message. If the network quality is insufficient to immediately complete server confirmation, the system generates a check-in record locally with driver identity, vehicle identity, timestamp, and GPS location, and marks the instruction as "locally completed, awaiting synchronization".

[0078] Subsequently, the system executes the second atomic business instruction. Since the task app does not provide a complete API, the system opens the task list page of the task app, uses OCR to identify the task number and title on the page, locates the second task, and simulates clicking to enter the task details page using system assistance functions. Upon entering the details page, the system identifies the destination of the second task as "Unloading Point No. 3," and uses this address as the navigation parameter for the third atomic business instruction, initiating navigation through the map app's URIScheme.

[0079] The output data in this embodiment includes: Composite instruction number; Three atomic business instructions and their execution order; The check-in result or pending synchronization status of the first atomic business instruction; The recognition results of the second task; The navigation destination of the third atomic business instruction; The current status and update time of each atomic business instruction; Synchronize the task status to the backend business system.

[0080] To verify the effectiveness of this embodiment, 600 real-life spoken compound commands from drivers were collected from 30 mining transport vehicles, including expressions such as "after arrival," "first...then...", and "and navigate." Using the traditional fixed command word scheme, only single-action commands could be correctly processed, with a compound command splitting success rate of 62.3%. Using a scheme that only matches keywords without considering the vehicle context, the compound command splitting success rate was 78.5%, but there was a problem of incorrectly executing "view task and navigate" in parallel. With the scheme of this embodiment, the system can reconstruct commands based on location conditions, task context, and application capabilities, achieving a compound command splitting success rate of 94.8%, and reducing the atomic command execution order error rate from 18.6% in the traditional scheme to 3.1%.

[0081] This embodiment demonstrates that the present invention does not simply identify keywords, but rather takes location conditions, sequential connections, business objects, and target application capabilities in natural language as input, and outputs an executable, traceable, and recoverable atomic business instruction chain, which can meet the actual execution requirements of composite operation instructions in industrial vehicle-machine scenarios.

[0082] An example: An implementation of intermediate state keep-alive for manual interruptions and weak network anomalies. In this embodiment, the vehicle is operating in an open-pit mine, and its current task is "to proceed to loading point number two for loading." The vehicle-mounted A11 application is in dual-mode operation (online / offline). The system receives a voice command from the driver: "After navigating to loading point number two, check in and then synchronize the current task status." The system receives the following input data: Natural language instruction text; Current vehicle location: 2.5 kilometers from loading point No. 2; The electronic fence coverage area of ​​loading point No. 2; Current task number; Driver's identity and vehicle identity; Network status: Initially the network was normal, but it entered a weak network area while driving; Map app status: Navigation is supported but may be manually terminated by the driver; Business system status: Both the attendance system and the task system support offline data transfer.

[0083] The system reconstructs the composite instruction into three atomic business instructions. The first atomic business instruction is "Navigate to Loading Point 2", the second atomic business instruction is "Check in upon arrival at Loading Point 2", and the third atomic business instruction is "Synchronize current task status after checking in". An intermediate state manager establishes a status record for each of the three atomic business instructions and writes it to a local state table. The state table includes at least the following fields: composite instruction number, atomic instruction number, current status, preconditions, completion conditions, target application, last update time, reason for exception, and whether compensation synchronization is required.

[0084] At the start of execution, the first atomic business instruction is in progress, and the map app begins navigation. During the journey, the driver, familiar with the route, closes the navigation page via touchscreen. The system detects the end of the map app navigation but does not immediately deem the entire composite instruction a failure; instead, it further reads the vehicle's current location. If the vehicle has not yet entered the electronic fence of loading point number two, the system prompts the driver: "Navigation has been canceled. Do you wish to retain automatic check-in upon arrival at loading point number two?" If the driver chooses to retain it, the first atomic business instruction is marked as "manually terminated," the second atomic business instruction remains in a waiting state, and the system continues to monitor the vehicle's location in the background. If the driver chooses to cancel the entire operation, both the second and third atomic business instructions are marked as canceled.

[0085] In this embodiment, the driver chooses to remain. The vehicle continues driving and enters the electronic fence of loading point number two. The system determines that the preconditions for the second atomic business instruction have been met and switches it from a waiting state to a ready state. At this time, the vehicle enters a weak network area in the mining area, and both the MQTT and WebSocket channels become unstable. When the system executes the check-in, it prioritizes generating a check-in record locally, including the driver's identity, vehicle identity, task number, loading point number, timestamp, and GPS location, and marks the status of the second atomic business instruction as "locally completed, pending synchronization". The third atomic business instruction "synchronize current task status" enters a suspended state due to network unavailability.

[0086] Once the vehicle leaves the weak network area, the communication management module detects network recovery. The intermediate state manager reads the local state table and prioritizes uploading the attendance record of the second atomic business instruction to the attendance system. Then, it synchronizes the task status of the third atomic business instruction to the backend business system. After successful synchronization, both the second and third atomic business instructions are marked as completed, and the composite instruction status changes to completed.

[0087] The output data in this embodiment includes: Manual termination record of navigation commands; The user's choice of whether to retain subsequent check-in instructions; The result of determining arrival at loading point number two; Local check-in records; Compensation and synchronization results after network recovery; The final task status in the backend business system.

[0088] To verify the effectiveness of intermediate state keep-alive, 100 composite commands including navigation, check-in, and task synchronization were selected as test samples. These included 30 instances of manual navigation termination, 40 instances of weak network or short-term network outages, and 30 instances of normal execution. Traditional solutions without intermediate state management typically end the entire process directly when navigation is terminated, resulting in a composite command completion rate of 68%. Solutions that only support offline retransmission but do not distinguish the state of atomic commands achieve a completion rate of 81%. With the solution in this embodiment, the system can distinguish between "navigation action terminated" and "whether the arrival condition is met," and can maintain subsequent check-in and synchronization tasks even under weak network conditions, achieving a composite command completion rate of 96%. Specifically, in the scenario of manual navigation termination, the success rate of retaining subsequent check-in tasks reached 93.3%; in the scenario of weak network conditions, the success rate of retransmitting critical business data reached 97.5%.

[0089] This embodiment does not use the success or failure of a single application action as the standard for the success or failure of the entire composite instruction. Instead, it continuously maintains the business conditions, user cancellation scope, and synchronization status of atomic business instructions through an intermediate state manager, so that composite instructions can continue to be kept alive and resume execution under common industrial anomalies such as manual interruption, weak network, and application switching.

[0090] An example: Spatiotemporal conflict resolution and output control of parallel composite tasks In this embodiment, the vehicle is located in a port logistics park, and its current primary task is to "proceed to loading area A to confirm container loading." Simultaneously, there is a queue query task for yard B in the business system. The driver inputs a voice command: "At the same time, go to loading area A to confirm the container loading and go to yard B to check the queuing situation." The input data received by the system includes: Natural language text; Current location of the vehicle; A. Loading area coordinates and electronic fence; B-yard coordinates and electronic fence; Current main task type: Confirm packing; Auxiliary task type: View queue; Current vehicle driving status; Target application support status: The packing confirmation APP requires the vehicle to arrive at loading area A for confirmation, and the queue query can be queried remotely through the business system or on-site; Network status: MQTT channel is healthy, WebSocket channel latency is high.

[0091] The system identifies "simultaneously" and "in parallel" as a parallel connection and initially breaks down the instruction into two atomic business instructions: First, confirm container loading at loading area A; second, check the queue status at yard B. Subsequently, the system performs conflict assessment based on vehicle location, task location, and business rules. The assessment determines that loading area A and yard B are approximately 3.8 kilometers apart, and the two locations cannot be reached simultaneously. "Confirm container loading" must be executed within the electronic fence of loading area A, while "check queue status" does not necessarily require reaching yard B if the business system can return queue data. Therefore, the system determines that the two atomic business instructions have a superficial parallel relationship, but the actual execution method needs to be rearranged based on the business execution conditions.

[0092] The system first identifies "Confirm Packing" as the key node of the current main task, with higher priority than "Check Queue Status". Simultaneously, the system checks if the business system supports remotely obtaining the queue status of yard B. If the business system returns queue data that can be viewed remotely, the system rewrites the second atomic business instruction as "Remotely Query Queue Status of Yard B", eliminating any spatial conflict with the first atomic business instruction. If the business system is unavailable or does not support remote querying, the system suspends the second atomic business instruction, waiting until the vehicle completes confirmation of packing at loading area A before prompting whether to proceed to yard B or continue the query.

[0093] In this embodiment, the business system supports remote querying of the queue status at yard B. The system outputs two reconstructed atomic business instructions: the first atomic business instruction is "Navigate to loading area A and confirm loading upon arrival," the target application is the loading confirmation APP, the precondition is that the vehicle enters the electronic fence of loading area A and the current task status is "Pending loading confirmation," and the completion condition is that the loading confirmation APP returns "Confirmation successful" or the vehicle terminal recognizes the words "Confirmed loading" on the page; the second atomic business instruction is "Query the queue status at yard B," the target application is the yard queuing system or scheduling APP, the precondition is that the business system supports remote queue query, and the completion condition is to obtain the current number of vehicles in the queue at yard B, the estimated waiting time, or the queue status.

[0094] The system further determines whether there is a spatiotemporal conflict between the two atomic business instructions. Since the first atomic business instruction requires the vehicle to actually arrive at loading area A before confirming the loading, while the second atomic business instruction in this embodiment can be completed remotely via the business system and does not require the vehicle to actually arrive at yard B, the system determines that there is no irreconcilable spatial conflict between the two atomic business instructions. The system sets the first atomic business instruction as the primary execution task and the second atomic business instruction as a task that can be remotely queried in parallel.

[0095] During execution, the system first launches the map application, sets the destination to loading area A, and enters navigation mode. Simultaneously, the system sends a queue query request for yard B to the yard queuing system via the vehicle-mounted communication module. If the network is normal, the system returns the queue status for yard B and displays it on the vehicle-mounted screen, for example: "Currently 12 vehicles are queued at yard B, estimated wait time 18 minutes." This query result does not affect the vehicle's continued navigation to loading area A.

[0096] When a vehicle enters the electronic fence of loading area A, the system checks if the current task status is still "Pending Packing Confirmation". If the task status is valid, the system launches the packing confirmation app. If the app provides a standard control interface, packing confirmation is completed via business messages; if the app does not provide a unified API, the system uses OCR to recognize the "Confirm Packing" control on the page and simulates a click to complete the confirmation using system auxiliary functions. After confirmation, the system updates the status of the first atomic business instruction to "Completed" and synchronizes the packing confirmation result to the backend business system.

[0097] To verify the technical effectiveness of this embodiment, comparative tests were conducted under the same vehicle, the same vehicle-mounted terminal, and the same mining area network environment. The test input command was "Go to loading area A to confirm the container loading, and at the same time check the queue status in yard B," and a total of 100 tests were conducted. Among them, when using the traditional single-command triggering method, the system could only recognize one of the actions, "navigate to loading area A" or "check the queue status in yard B," and the composite command was successfully executed 61 times. When using the simple parallel distribution method, the system did not distinguish between on-site confirmation tasks and remote query tasks, resulting in task conflicts, page overwriting, or incorrect execution order, and the number of successful executions was 74 times. After adopting the composite intent reconstruction and conflict resolution method of this embodiment, the system was able to distinguish between the container loading confirmation command that must be executed on-site and the queue query command that can be executed remotely, and the number of successful executions was 96 times.

[0098] This embodiment does not simply mechanically break down parallel semantics such as "simultaneously" and "and" and execute them concurrently. Instead, it further combines the target application's execution conditions, the vehicle's current location, the geofence of the work site, the task status, and the remote capabilities of the business system to determine whether there are genuine spatiotemporal conflicts among the atomic business instructions. For instructions that must be completed on-site, the system binds them to vehicle arrival conditions; for instructions that can be completed remotely, the system allows them to be executed in parallel with navigation or on-site confirmation tasks. Thus, without increasing the driver's workload, the system improves the execution completeness of complex industrial operation instructions and the reliability of the business loop.

[0099] In another embodiment, the present invention provides an in-vehicle terminal control device. The device includes a mode switching module, an instruction parsing module, a communication management module, and an execution control module. The mode switching module runs in the background Service of the A11 application on the vehicle's infotainment system. It monitors the network connection status in real time by calling the TelephonyManager interface and automatically switches between an online mode based on cloud parsing and an offline mode based on local instruction templates based on the monitoring results. The instruction parsing module is deployed on the MCP control server and performs multi-level parsing on received natural language instructions based on preset matching rules to identify the target application identifier, instruction type, and control parameters. The communication management module is responsible for maintaining a long-connection channel based on the MQTT protocol and a real-time full-duplex channel based on the WebSocket protocol; high-priority job instructions are transmitted concurrently through dual channels, while ordinary instructions are dynamically switched through a single channel; and the target channel is dynamically selected to transmit the parsed instructions based on the real-time monitored channel health status (such as latency and packet loss rate). The execution control module is installed on the vehicle's infotainment system and is used to sequentially detect the target application's support for the control interface according to the priority order of URI Scheme, broadcast, and system auxiliary functions, and select the first supported control method to execute the instruction. This device uses a modular design to decouple complex logic, thereby improving the maintainability and scalability of the system.

[0100] In another embodiment, the present invention provides an in-vehicle terminal control system. The system includes a memory and a processor. The memory uses 128GB of eMMC 5.1 flash memory and stores a computer program. The processor uses a Rockchip RK3588 in-vehicle chip; when the processor executes the computer program in the memory, it implements the aforementioned in-vehicle terminal control method. The system also includes a 4GB LPDDR4 RAM, expandable with multiple or higher RAM modules to support the rapid computation of multi-level parsing algorithms. This hardware configuration ensures the real-time performance of instruction parsing and execution, and its physical connection is achieved through a high-speed bus on the PCB board, with a data transmission rate meeting the requirements of real-time in-vehicle control.

[0101] In another embodiment, the present invention provides a detailed definition of the multi-level parsing process. Multi-level parsing includes: identifying the target application identifier using a first-level regular expression match, for example, using the regular expression "^open(?P <app>Extract the application name the user wants to operate on using "and"; use a second-level regular expression to identify the command type, for example, using the regular expression "and(?P <action>The specific geographical location or time parameters are extracted using the .?)(” method. Compared to complex deep learning models, this three-level regular expression matching mechanism has extremely high execution efficiency and interpretability. It runs very fast on embedded vehicle terminals or edge computing gateways, with a parsing latency typically less than 10ms (parsing time was tested under instruction sets of different complexities to ensure that parsing long instructions does not cause interface lag).

[0102] In another embodiment, the present invention specifically selects the communication channels. At least two communication channels with different protocol characteristics include a long-connection channel based on the MQTT protocol and a real-time duplex channel based on the WebSocket protocol; high-priority job instructions are transmitted concurrently through both channels, while ordinary instructions are dynamically switched between single channels; the MQTT channel adopts QoS 1 (at least once) to ensure that instructions are not lost during network fluctuations, and its heartbeat interval is set to 20-60 seconds. The WebSocket channel uses the standard WSS encryption protocol, with port number 8081, for transmitting screen coordinate data requiring high-frequency interaction; through this dual-protocol redundancy design, the system can utilize the low bandwidth consumption of MQTT to maintain long connections while leveraging the high real-time performance of WebSocket for fine-grained operations, achieving a balance between communication efficiency and reliability.

[0103] In another embodiment, the present invention refines the logic for dynamically selecting the target channel. The system monitors the latency, packet loss rate, and heartbeat response status of each communication channel in real time. Specifically, the system sends a probe packet to the server every 5 seconds and calculates the round-trip time (RTT). When the latency of the current primary channel (such as MQTT) exceeds 200ms or the packet loss rate exceeds 5%, the system automatically switches to a backup channel (such as WebSocket) and performs state synchronization. State synchronization includes synchronizing the sequence number of currently incomplete instructions, execution progress, and cached UI node information to the new channel, ensuring that the switching process is imperceptible to the business logic. This dynamic switching mechanism based on real-time indicators greatly improves the system's survivability in complex geographical environments such as mining areas and tunnels.

[0104] In another embodiment, the present invention specifies the selection priority of multiple preset control modes. The selection priority is as follows: URI Scheme control mode is higher than broadcast control mode, and broadcast control mode is higher than simulated interactive operation based on system auxiliary functions. When executing control, the execution control module first queries the PackageManager to see if the target application has registered a specific Intent Filter. If it exists, it is given priority to invoke the application via URI Scheme and pass parameters; if this fails, it attempts to send a system broadcast with a specific Action; if the target application still does not respond, it finally starts the AccessibilityService to simulate clicks at the screen pixel level. This priority strategy reflects the design concept of "native interface priority, simulated operation as a backup," which ensures the success rate of control while minimizing the consumption of system resources and interference with the user interface.

[0105] In another embodiment, the present invention adds a data cleaning step before multi-level parsing. The data cleaning module standardizes the original instruction text, including removing preset non-feature characters (such as spaces and special punctuation), filtering interjections (such as "uh," "that," and "please help me"), and unifying the text format (such as converting Chinese numerals to Arabic numerals). For example, "Please help me open that driver assistant and clock in" is cleaned to "Open driver assistant and clock in". The data cleaning uses a dictionary-based replacement algorithm, with a processing time of less than 1ms; this step significantly improves the accuracy of subsequent regular expression matching and avoids parsing failures caused by colloquial expressions.

[0106] In another embodiment, the present invention adds a security verification step after multi-level parsing. The security verification module first checks the compliance of call permissions based on the target application identifier to determine whether the user has the right to operate the target APP (e.g., financial APPs require secondary authorization). Next, it limits the frequency of command calls based on a preset time window, for example, setting that the same application cannot receive more than 3 control commands within 10 seconds. The verification logic records the call frequency through Redis. The security verification ensures that the system is not attacked by malicious scripts and also prevents accidental operations caused by voice-triggered commands, thus ensuring system safety during driving.

[0107] In another embodiment, the present invention provides a data compensation step. In offline mode, the vehicle-mounted A11 application adds instructions that cannot be executed immediately (such as complex instructions requiring cloud computing power) to a local execution queue. This queue is stored in a local encrypted database and has the characteristic of not losing data even when power is off. When network recovery is detected (e.g., RSRP rises above -90dBm) and the system switches to online mode, the system automatically starts a synchronization thread to synchronize the instructions in the local execution queue to the server or business system for supplementary processing. The synchronization process uses a breakpoint resume mechanism, and each successfully synchronized instruction is deleted from the local queue. This step solves the problem of business interruption when the vehicle passes through a signal blind spot, ensuring the final consistency of critical business data such as "work check-in".

[0108] In another embodiment, the present invention implements a business collaboration step. The vehicle-mounted terminal synchronizes the application operation status of the vehicle terminal to the backend business management system in real time via specific protocol messages (such as custom Protobuf format messages; compared to JSON, Protobuf can reduce traffic consumption by approximately 40%). Status information includes the current foreground APP package name, the current page Activity name, and the ID of the last clicked control, etc. After receiving the status synchronization, the backend business management system triggers the corresponding business logic, such as automatically completing attendance check-in records, pushing the next transportation task to the driver, or updating the real-time operation status of the vehicle. This end-to-cloud collaboration mechanism transforms the vehicle terminal from an isolated island into a real-time online node in the enterprise's digital management system, greatly improving the automation level of fleet management.

[0109] In another embodiment, the execution logic of the execution control module is as follows: Following the priority order of URI Scheme, broadcast, and system accessibility, the module sequentially probes the target application's support for the control interface. The probe process includes: first, attempting to call the PackageManager.queryIntentActivities method; if the returned list is not empty, it determines that the URI Scheme is supported; if empty, it attempts to check whether a Receiver for receiving a specific Action is defined in the target app's Manifest file using reflection; if still not found, it determines that system accessibility is required. After selecting the first supported control method, the execution control module encapsulates the corresponding execution parameters and calls the system API. For example, if the accessibility method is selected, it calculates the absolute coordinates of the target button at the current screen resolution (e.g., 1280x720) and triggers a simulated click. This probe mechanism ensures that control commands are received and executed by the target application in the most reliable and fastest way.

[0110] In one embodiment, the system includes: reconstructing the compound intent based on conditional connectives; the vehicle-mounted terminal receives the driver's voice command: "After arriving at the loading point, check in first, then check the second task and navigate there." After the system converts the voice into text, it first performs standardization processing, removing non-business words such as "help me" and "a little," and retaining connective expressions such as "after arriving at...", "first", "then", and "and".

[0111] The vehicle-mounted system identifies "loading point" as the location condition, "check-in," "view task," and "navigation" as the work actions, and "second task" as the task object based on the local business vocabulary. Among these, "after arriving at the loading point" is identified as a conditional trigger relationship, "first" and "then" are identified as a sequential execution relationship, and "and" in this scenario is identified as a continuous execution relationship with data dependency on the previous action, meaning that the second task must be viewed first and the destination must be obtained before navigation can begin.

[0112] Based on this, the system reconstructs the original composite instruction into three atomic business instructions: First, the vehicle performs a check-in upon entering the loading point's electronic fence; second, after successful check-in, the task application is opened and the second task is viewed; third, the destination is extracted from the second task and navigation is initiated. Each atomic business instruction records the target application, preconditions, completion conditions, and exception handling method.

[0113] If the vehicle has not yet arrived at the loading point, the first atomic business instruction enters a waiting state, and the system continuously determines whether the triggering conditions are met based on GPS or business system location. If the vehicle arrives at the loading point but the network is abnormal, the check-in action can first generate a record to be synchronized locally with driver identity, vehicle identity, time, and location, and upload it after the network is restored. If the second task cannot be read through the interface, the system switches to OCR screen recognition mode, locates the second task in the task list, and enters the details page.

[0114] Technical effect: It solves the problem of scattered isolated actions such as simple execution of "check-in", "view task" and "navigation". By combining industrial operation location, task context and conjunction relationship, it reconstructs the driver's colloquial compound instructions into atomic instruction chain with preconditions and data dependencies, so as to be applicable to complex vehicle-machine scenarios such as mining and logistics.

[0115] In one embodiment, the system includes: intermediate state keep-alive in the face of abnormal interruption; the driver inputs: "Navigate to the loading point, clock in, and then synchronize the task status." The system reconstructs this compound instruction into three atomic business instructions: navigate to the loading point, clock in upon arrival, and synchronize the task status after clocking in, and the intermediate state manager records the current status of each instruction.

[0116] When the first atomic business instruction, "Navigate to loading point," is in progress, if the driver actively terminates the navigation via touchscreen or voice, the system does not immediately cancel the entire composite task. Instead, it determines whether the vehicle's current location meets the subsequent check-in conditions. If the vehicle has entered the loading point's electronic fence, the system marks the navigation instruction as "Manually terminated but arrival conditions met" and switches the check-in instruction to the ready state, continuing the check-in process. If the vehicle has not reached the loading point, the system prompts the driver to choose either "Cancel navigation only" or "Cancel the entire operation process."

[0117] When a driver selects "Cancel navigation only," subsequent check-in instructions remain pending, and the system continues to monitor whether the vehicle has arrived at the loading point. When a driver selects "Cancel the entire operation," both check-in and task synchronization instructions are marked as canceled, and the cancellation reason is recorded in the local status table. If the network is unavailable at this time, the status table is saved locally; after the network is restored, the cancellation or completion result is synchronized to the backend business system.

[0118] Each atomic business instruction has at least the following statuses: generated, waiting, ready, executing, suspended, completed, canceled, and failed. The status table records the instruction number, the associated compound instruction, the target application, the preconditions, the completion conditions, the last update time, the reason for the exception, and whether synchronization is required.

[0119] Technical effect: In the industrial vehicle scenario, where "the preceding action is manually terminated but the subsequent business conditions may still be met", the intermediate state manager distinguishes between the navigation action result and the business arrival condition, avoids the accidental cancellation of compound tasks, and achieves long-term keep-alive and recoverable execution of the work process.

[0120] In one embodiment, the process includes: resolving spatiotemporal conflicts between parallel tasks; the driver inputs: "Simultaneously go to loading point one to confirm loading, and go to loading point two to check the queue." The system recognizes "simultaneously" and "and" as parallel connections and splits them into two atomic business instructions: first, go to loading point one to confirm loading; second, go to loading point two to check the queue.

[0121] Before execution, the system reads the vehicle's current location, the location of loading point one, the location of loading point two, the current task stage, and the business priority. If the two loading points are far apart, and both confirming loading and checking the queue require the vehicle to reach the corresponding location, the system determines that the two atomic business instructions have a spatial conflict and cannot be executed in parallel. At this time, the intermediate state manager transfers the two instructions to the conflict determination state.

[0122] The system resolves requests based on business rules: If "Confirm Loading" is a critical node of the current main task, while "Check Queue" is a secondary query action, then confirming loading will be executed first, the "Check Queue" command will be suspended, and the driver will be notified that "the current main task of confirming loading has been executed first; checking the queue at loading point number two will be executed after the conditions are met." If both commands are query actions, the task closer to the vehicle's current location will be executed first, and the other task will enter a waiting state. If both tasks are high priority and the system cannot automatically determine the priority, the driver will be prompted to select the priority task.

[0123] When a suspended instruction meets the recovery conditions—for example, when a vehicle proceeds to loading point two after confirming loading point one, or when the business system returns that the queue status can be viewed remotely—the system switches the instruction from the suspended state to the ready state and continues execution. If the recovery conditions are not met after a preset retention time, the instruction is marked as not executed and the reason is synchronized.

[0124] Technical effect: For parallel tasks in the compound instructions of industrial vehicles, the system not only recognizes the semantic "simultaneous execution", but also combines vehicle position, operation location, main task priority and business execution conditions to resolve spatiotemporal conflicts, thereby ensuring that key operation instructions are executed first and avoiding invalid or conflicting operations.

[0125] This technical solution is mainly applied to the remote intelligent control scenario of vehicle-to-vehicle systems in intelligent connected vehicles, especially targeting business scenarios in vertical industries such as mining transportation and freight logistics, and specifically includes the following sub-scenarios: Scenario 1: Driver's voice control of the vehicle's infotainment system and business systems During driving, drivers can issue natural language commands through voice assistants (such as Xiaofeng AI) to achieve deep integration between vehicle control and business systems, for example: "Navigate to XXA" - The system automatically plans the route and synchronizes it to the freight task system. "Navigate from location XXB to location XXC" - Supports multi-point route planning, suitable for freight delivery scenarios. "Play XXX's Confession Balloon" - Music Playback Controls "Open the E11 platform and scan" - Quick QR code scanning function "Open Driver Assistant" - Open the non-standard mining APP, which supports voice-controlled check-in function. "I need to clock in" - Automatically clocking in for work via MQTT message triggers the mining app. "Open Freight Assistant" - Open the non-standard freight APP to view the current task. "View Task Details" - Automatically opens the freight app and redirects to the task details page. Π View the second task Π - Support opening specified task details by sorted index The system obtains text instructions through speech recognition, converts them into structured control instructions after natural language parsing, sends them to the vehicle terminal for execution through the dual-channel communication network, and implements data interconnection with back-end business systems (such as mine management system, logistics scheduling system or attendance management system).

[0126] Scenario 2: Remote pre-operation of vehicle terminal and business collaboration The user remotely controls the vehicle terminal through Xiaofeng Intelligent Agent and collaborates with the business system, for example: Set the navigation destination remotely before departure, and the system automatically associates the freight task Remotely open the in-vehicle freight APP and load task data in advance Remotely adjust the volume of the vehicle terminal Remotely trigger the check-in process - before the driver arrives at the mining area, complete the check-in preparation in advance by voice Scenario 3: Push of freight tasks and linkage with navigation When the dispatching system issues a new freight task: The task notification is pushed to the vehicle terminal in real time through the MQTT channel The vehicle terminal automatically opens the freight APP and displays the to-do task The driver can quickly view task information by voice "view task details" Click the navigation button to start navigation to the loading point / unloading point with one click The navigation destination is automatically associated with the freight task to realize a closed business loop Scenario 4: Local control in offline mode When the vehicle is in an area with weak or no network signal (such as underground parking lots, remote mountainous areas, and inside mines), the system switches to offline mode, and executes common control functions through local instruction template matching, such as music playback control and volume adjustment. After the network is restored, the instruction records during the offline period are automatically synchronized to the business system.

[0127] Scenario 5: Multi-device collaborative control The user controls the vehicle terminal through Xiaofeng Intelligent Agent to realize the linkage between the vehicle terminal and the smart home, for example: Set the vehicle terminal navigation to the first delivery point tomorrow through Xiaofeng Intelligent Agent at home The vehicle terminal is linked with the smart home scene, for example, "I am going out" automatically triggers the vehicle terminal preheating and opens the freight APP 3.1 System architecture embodiment The system architecture of the present technical solution includes the following core modules: (1) MCP control server The control service, deployed in the cloud and implemented using Python and the MCP SDK, provides the following functionalities: Device Management: Maintain the mapping relationship between Xiaofeng intelligent agent MAC and vehicle system SN. Command parsing: Invoking cloud-based NLP services to parse natural language commands. Dual-channel scheduling: Selecting the optimal communication channel based on the vehicle-mounted system's channel status. MQTT processing: Communicating with the MQTT Broker via the Paho MQTT client. WebSocket service: Provides WebSocket service based on the websockets library. (2) A11 application on the vehicle terminal The A11 application deployed on the vehicle's infotainment system includes the following components: RemoteControlService: Dual-channel remote control service MqttControlClient: MQTT client implementation WebSocketClientAdapter: WebSocket client adapter CommandParser: Natural Language Command Parser CarAppController: Vehicle Application Controller OfflineCommandProcessor: Offline command processor SecureCommandChannel: Secure Communication Channel (3) MQTT Broker The message middleware, using HiveMQ or Eclipse Mosquitto, is responsible for: routing and forwarding command messages; offline message persistence (QoS level 1); and maintaining device online status. 3.2 Key Code Examples An example: Natural language instruction parsing Example 2: Dual-channel adaptive communication Example 3: Multimodal Application Control Example 4: Secure Communication Channel 3.3 Experimental Data To verify the effectiveness of this technical solution, the following experimental tests were conducted: Experiment 1: Natural Language Command Parsing Accuracy Test Test sample: 1000 natural language commands (covering navigation, music, application control, volume control, and other scenarios) Navigation instructions 300 291 97.0% Music Commands 250 242 96.8% Application Control 250 238 95.2% Volume control 200 194 97.0% overall 1000 965 96.5% Experiment 2: Dual-channel communication reliability test Test scenario: Simulating network fluctuations (packet loss rate 5%, latency 200ms) Single MQTT channel 1000 923 92.3% 156ms Single WebSocket Channel 1000 935 93.5% 142ms Dual-channel adaptive 1000 994 99.4% 138ms Experiment 3: Application Control Coverage Test Test subjects: Six mainstream in-vehicle infotainment applications, including A11 Map, B11 Map, C11 Music, D11 Cloud Music, E11 Platform, and F11 Platform. Experiment 4: Offline Mode Usability Test Test scenario: Vehicle network connection turned off Experimental conclusion: The proposed technical solution is significantly superior to existing technologies in terms of natural language understanding accuracy, communication reliability, and application control coverage.

[0128] This invention and its technical solution can also be extended to the following technical fields: smart mining / port field: voice control of mining transport vehicles and port operation vehicles linked with production systems; smart logistics field: voice control of freight vehicles and integration with logistics scheduling systems; industrial equipment remote control field: remote monitoring and operation of equipment in industrial scenarios; mobile terminal smart assistant field: enhanced voice assistant functions of smartphones, tablets and other devices; IoT device management field: unified access and intelligent control of various IoT terminal devices.

[0129] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. Other modifications can be easily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.< / action> < / app>

Claims

1. A method for in-vehicle infotainment applications based on screen recognition and command reconstruction, characterized in that, include: S10, based on the network connection status monitoring results, automatically switches between online mode based on cloud parsing and offline mode based on local command templates; S20: Based on preset matching rules, the received natural language instructions are parsed at multiple levels to identify the target application identifier, instruction type and control parameters; S30, maintain at least two communication channels with different protocol characteristics, and dynamically select the target channel to transmit the parsed instructions based on the real-time monitored channel health status; S40, based on the target application's support for the control interface, select the target control mode from multiple preset control modes to execute the instruction; wherein, the multiple preset control modes include simulated interactive operations based on system auxiliary functions as the underlying control.

2. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, S20， The multi-level parsing includes: using a first-level regular expression to identify the target application identifier, using a second-level regular expression to identify the instruction type, and using a third-level regular expression to extract the control parameters from the natural language instruction.

3. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, S30, the at least two communication channels with different protocol characteristics include a long connection channel based on the MQTT protocol and a real-time full-duplex channel based on the WebSocket protocol; high-priority job instructions are transmitted concurrently through dual channels, while ordinary instructions are dynamically switched through a single channel.

4. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, S30, the dynamic selection of target channel includes: real-time monitoring of the latency, packet loss rate and heartbeat response status of each communication channel; when the latency or packet loss rate of the current main channel exceeds a preset threshold, automatically switching to the backup channel and synchronizing the status; multi-indicator weighted adaptation to industrial vehicle status.

5. A vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, S40, the selection priority of the various preset control modes is as follows: URI Scheme control mode is higher than broadcast control mode, broadcast control mode is higher than business message control mode, and business message control mode is higher than simulated interactive operation based on OCR screen recognition, dynamic coordinate scaling and system auxiliary functions.

6. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, Before the multi-level parsing, a data cleaning step is also included: standardizing the original instruction text, including removing preset non-feature characters, filtering interjections, and unifying the text format.

7. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, Following the multi-level parsing, a security verification step is also included: performing a compliance check on the call permission based on the target application identifier, instruction type, vehicle identity, and user permissions, and limiting the frequency of instruction calls from the same user, the same vehicle, or the same in-vehicle terminal based on a preset time window.

8. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, It also includes a data compensation step: in the offline mode, structured business control instructions that cannot be executed or reported in real time are added to the local execution queue, and conflict resolution is performed according to the instruction timestamp and the business priority queue; when the network is detected to be restored and the system switches to the online mode, the structured business control instructions in the local execution queue are synchronized to the server or business system.

9. The in-vehicle infotainment application method based on screen recognition and command reconstruction according to claim 1, characterized in that, It also includes business collaboration steps: the application operation status of the vehicle terminal is synchronized to the backend business management system in real time through specific protocol messages to trigger corresponding business check-in, task push, task confirmation, vehicle queuing or navigation linkage operations.

10. A vehicle infotainment system based on screen recognition and command reconstruction, characterized in that, include: The mode switching module is used to automatically switch between online mode based on cloud parsing and offline mode based on local command templates based on network connection status monitoring results. The instruction parsing module is used to perform multi-level parsing of received natural language instructions based on preset matching rules in order to identify the target application identifier, instruction type and control parameters; The communication management module is used to maintain at least two communication channels with different protocol characteristics, and dynamically select the target channel to transmit the parsed instructions based on the real-time monitored channel health status. The execution control module is used to select the target control mode from a variety of preset control modes to execute the instruction based on the target application's support for the control interface. The variety of preset control modes includes simulated interactive operations based on system auxiliary functions as the underlying control.

Images