Screen projection control method and device based on process life cycle security management

Abstract

This invention discloses a screen projection control method and apparatus based on process lifecycle security management, relating to the field of vehicle-mounted terminal screen projection control technology. The method includes: configuring a first sub-process layer compatible with the user terminal system program, and starting a second sub-process layer that runs a screen projection bridging process; adding the process of the second sub-process layer to the global security management list; parsing the current operating environment to obtain level resolution information; depending on whether the level resolution information is non-low performance or low performance, parsing the video stream through the screen projection bridging process or starting a screenshot process to capture and display the screen; obtaining the instruction execution mode corresponding to the level resolution information according to mode selection rules, and responding to touch operations. This application achieves system compatibility and smooth screen projection by using a first sub-process layer compatible with the user terminal system program and a second sub-process layer running the screen projection bridging process, and improves the stability of screen projection control through process lifecycle security management.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted terminal screen projection control technology, and in particular to a screen projection control method and device based on process lifecycle security management. Background Technology

[0002] With the gradual popularization and development of domestic operating systems, more and more smart terminals (including in-vehicle terminals, tablets, smart screens, and mobile phones) are equipped with domestic operating systems (such as HarmonyOS). However, in software testing and screen projection control scenarios, there is an urgent need for the ability to remotely project screens onto smart terminals equipped with corresponding operating systems. Especially in the scenario of automated testing of in-vehicle systems, a tool is needed that can replace manual operation and realize remote control and automated UI verification. The current practical application faces the following difficulties: different operating systems have their own different testing tools, which cannot be effectively compatible in screen projection control; and the underlying communication protocols and shell command systems of different operating systems are significantly different, and existing Android tools cannot be adapted to terminal devices equipped with other operating systems. There are very few screen projection control solutions for non-Android operating systems on the market, and existing official tools (such as DevEco Studio) only support manual debugging by developers and do not have a programmable Python API interface, making it difficult to integrate into existing automated testing processes. Existing technical methods have unreliable compatibility issues in screen projection control applications between terminals with different operating systems. Summary of the Invention

[0003] This invention provides a screen mirroring control method and apparatus based on process lifecycle security management, aiming to solve the problem of unreliable compatibility in screen mirroring control applications between terminals with different operating systems in existing technologies.

[0004] In a first aspect, embodiments of the present invention provide a screen projection control method based on process lifecycle security management, wherein the method is applied in an in-vehicle terminal, the in-vehicle terminal communicating with a user terminal to achieve data information transmission, and the method includes: If a screen-projected file is received from the user terminal, a first sub-process layer is configured, which is compatible with the system program of the user terminal. The second subprocess layer is started through the first subprocess layer, and the screen casting bridging process is run in the second subprocess layer; All processes running in the second child process layer are added to a pre-defined global security management list to achieve process security management; The current operating environment is analyzed according to the preset environment level analysis rules to obtain the corresponding level analysis information; If the grade resolution information is not a low performance grade, the video stream corresponding to the screen casting file is obtained through the first sub-process layer and the screen casting bridging process and then displayed. If the performance level is low, a screenshot process is started in the first sub-process layer to capture the corresponding display screen from the projection file for display. If a touch command is received from the user terminal, the command execution mode corresponding to the level parsing information is obtained according to the preset mode selection rules; The touch instruction is parsed according to the instruction execution mode and the touch operation corresponding to the touch instruction is responded to in the video stream or the display screen.

[0005] Secondly, embodiments of the present invention also provide a screen projection control device based on process lifecycle security management, wherein the device is configured in an in-vehicle terminal, the in-vehicle terminal communicates with a user terminal to realize data information transmission, and the device is used to execute the screen projection control method based on process lifecycle security management as described in the first aspect above, the device comprising: The first subprocess layer configuration unit is used to configure the first subprocess layer if a screen projection file is received from the user terminal. The first subprocess layer is compatible with the system program of the user terminal. The process execution unit is used to start the second sub-process layer through the first sub-process layer and run the screen projection bridging process in the second sub-process layer. The process addition unit is used to add all processes running in the second sub-process layer to a preset global security management list to achieve process security management; The level resolution information acquisition unit is used to parse the current operating environment according to the preset environment level resolution rules and obtain the corresponding level resolution information; The video stream display unit is used to parse the video stream corresponding to the projection file through the first sub-process layer and the projection bridging process and display it if the level parsing information is not low performance level. The display screen unit is used to start a screenshot process in the first sub-process layer to capture the corresponding display screen from the projection file and display it if the level parsing information is a low performance level. The instruction execution mode acquisition unit is used to acquire the instruction execution mode corresponding to the level parsing information according to the preset mode selection rules if a touch instruction is received from the user terminal. A touch operation response unit is used to parse the touch instruction according to the instruction execution mode and respond to the touch operation corresponding to the touch instruction in the video stream or the display screen.

[0006] Thirdly, embodiments of the present invention also provide a computer device, wherein the device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the screen projection control method based on process lifecycle security management described in the first aspect above.

[0007] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the screen projection control method based on process lifecycle security management as described in the first aspect above.

[0008] This invention provides a screen projection control method and apparatus based on process lifecycle security management, relating to the field of vehicle terminal screen projection control technology. The method includes: configuring a first sub-process layer compatible with the user terminal system program, and starting a second sub-process layer that runs a screen projection bridging process; adding the process of the second sub-process layer to a global security management list; parsing the current operating environment to obtain level resolution information; depending on whether the level resolution information is non-low performance or low performance, parsing the video stream through the screen projection bridging process or starting a screenshot process to capture and display the screen; obtaining the instruction execution mode corresponding to the level resolution information according to mode selection rules, and responding to touch operations. This application achieves system compatibility and smooth screen projection by using a first sub-process layer compatible with the user terminal system program and a second sub-process layer running the screen projection bridging process, and improves the stability of screen projection control through process lifecycle security management. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating the screen projection control method based on process lifecycle security management provided in this embodiment of the invention; Figure 2 A schematic diagram illustrating an application scenario of the screen projection control method based on process lifecycle security management provided in this embodiment of the invention; Figure 3 A schematic block diagram of a screen projection control device based on process lifecycle security management provided in an embodiment of the present invention; Figure 4This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0013] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] This invention application provides a screen projection control method based on process lifecycle security management. This method is applied to an in-vehicle terminal, where the in-vehicle terminal executes a stored software program to implement the aforementioned screen projection control method based on process lifecycle security management. For example... Figure 2 As shown, the vehicle-mounted terminal 10 communicates with the user terminal 20 to transmit data. The vehicle-mounted terminal 10 is a terminal device configured on the vehicle side for vehicle control and multimedia playback, such as a vehicle-mounted tablet computer with an MCU chip. The vehicle-mounted terminal runs a program system to drive the hardware modules to achieve corresponding functions. The vehicle-mounted terminal 10 includes at least a processing unit, a display unit, and a touch screen. The user terminal 20 is a terminal device operated by users or testers to achieve screen projection connection with the vehicle-mounted terminal 10. The user terminal 20 can be a laptop, tablet computer, desktop computer, or mobile phone, etc. Figure 1 As shown, the method includes steps S110 to S180.

[0016] S110. If a screen-projected file is received from the user terminal, configure a first sub-process layer, which is compatible with the system program of the user terminal.

[0017] The user terminal is a terminal device running the HarmonyOS operating system. After the in-vehicle terminal receives the screen-projected file from the user terminal, it first establishes a runtime environment capable of communicating with the user terminal's system program. This process is achieved by configuring the first sub-process layer, which corresponds to the HDC (HarmonyOS Device Connector) sub-process layer. The HDC sub-process layer supports pure Python. The first sub-process layer is entirely implemented based on the Python standard library, requiring no Java Runtime Environment (JRE), making it suitable for automated script execution, CI / CD (Continuous Integration / Continuous Delivery) pipelines, and environments without a JRE. As the basic command sub-process layer, the first sub-process layer adopts an interface standard compatible with the user terminal's system program and is primarily responsible for calling system command-line tools to perform basic operations. The first sub-process layer is entirely based on the standard library, requiring no additional runtime environment, making it suitable for automated script execution and scenarios without a specific runtime environment. The first sub-process layer occupies a fundamental position in the architecture, bearing the responsibility of interacting with the underlying system.

[0018] For example, the first subprocess layer can use the Python subprocess module (a built-in module in Python that creates subprocesses within Python code, calls external executable programs, executes operating system commands, and enables inter-program calls) to call the HDC command-line tool and execute shell (command interpreter) commands to perform various basic operations. Specific operations include: injecting multi-touch events using the "uinput -M" command; capturing the device screen as a JPEG file (a common static image compression format) using the "snapshot_display -f" command and saving it locally using "file recv"; exporting the UI (user interface) hierarchy tree (JSON format, JavaScript Object Notation, a lightweight text data format often used to describe structured data) using the "uitest dumpLayout" command; injecting keyboard key events using the "uinput -K" command; and inputting Chinese characters and other Unicode (Unified Character Encoding Standard, encompassing almost all characters worldwide, including Chinese, English, and symbols, and supporting Chinese input) text using the "uitest uiInput text" command. By configuring a compatible first subprocess layer, the vehicle terminal can provide a stable underlying communication foundation for subsequent screen projection data processing.

[0019] S120. Start the second sub-process layer through the first sub-process layer, and run the screen casting bridge process in the second sub-process layer.

[0020] After the first sub-process layer is configured, the second sub-process layer is started through this layer, and the screen mirroring bridging process runs in the second sub-process layer. The screen mirroring bridging process is a key component for implementing high-performance video stream relay. It runs in an independent second sub-process layer to avoid blocking the main process. The screen mirroring bridging process uses the underlying Software Development Kit (SDK) to establish a connection with the user terminal and is responsible for real-time video frame acquisition and touch event relay. Specifically, the screen mirroring bridging process can be a Java StreamBridge process (which acts as an intermediate bridge program to forward video streams and touch commands). This process is a Java process, and it uses the HarmonyOS HOScrcpySDK (HarmonyOS version of the screen mirroring development kit) to establish a socket connection with the user terminal to achieve real-time video frame acquisition and touch event relay.

[0021] For example, the screen mirroring bridging process can obtain screen image data by calling the debugging interface provided by the user terminal, or receive touch events from the user terminal by establishing a dedicated data transmission channel. The screen mirroring bridging process implements protocol-based communication between the second and first sub-process layers, specifically including: frame output (stdout): each frame uses a binary protocol of "4-byte big-endian length prefix + data body," where the data body is a JPEG compressed image or raw H.264 NAL units (controlled by the raw_mode parameter); touch command input (stdin): uses a text line protocol of "D:x:y\n", "M:x:y\n", and "U:x:y\n", representing touch down, move, and up, respectively; status log output (stderr): the Java process outputs runtime logs and ready signals ("READY" keyword) through stderr (Standard Error stream). This two-layer architecture separates basic command calls from high-performance video stream relay, ensuring compatibility of basic functions while providing an implementation path for high-performance mode.

[0022] S130. Add all processes running in the second subprocess layer to the preset global security management list to achieve process security management.

[0023] To ensure system stability and resource security during screen mirroring, all processes running in the second sub-process layer need to be added to a pre-defined global security management list. The global security management list is a data structure used for centralized management and monitoring of all screen mirroring-related processes. Maintaining this list enables secure management of process lifecycles. The global security management list is the core carrier for process registration and deregistration; for example, it could be the `_ACTIVE_PROCS` global list. For instance, all Java sub-processes started in the second sub-process layer are added to the `_ACTIVE_PROCS` global list using the `_register_proc()` function (protected by the `threading.Lock` mutex) to achieve secure process management; when a process terminates, it is removed using `_unregister_proc()`. When any process in the second sub-process layer starts, the system automatically registers its process identifier (PID) or handle to the global security management list. If a process in the global security management list terminates abnormally or exits normally, the system removes it from the list. Furthermore, the global security management list also supports triggering cleanup operations when the main process exits, traversing the list and terminating all remaining sub-processes, effectively preventing process leaks and resource residue. For example, a global security management list can be implemented using a hash table or linked list data structure to ensure the efficiency of process addition and removal operations; it can also be combined with a process generation counter to prevent older cleanup threads from mistakenly killing newly started processes.

[0024] In a specific embodiment, step S130 includes the following sub-steps: if a process in the global security management list terminates, the terminated process is removed from the global security management list; if the main process in the first child process layer exits normally, all processes in the global security management list are terminated and removed; it is determined whether the current generation of the process in the global security management list matches the counter generation; if the current generation of the process does not match the counter generation, the process is terminated and removed from the global security management list.

[0025] During the operation of the vehicle terminal, Java processes within the second sub-process layer may terminate due to task completion, exceptions, or insufficient resources. To prevent data bloat and misjudgments caused by retained invalid process references in the global security management list, the liveness status of each process in the list needs to be continuously or periodically monitored. This process can be implemented through the process status query interface provided by the operating system to detect whether a process has become a zombie or has completely terminated. Once a process is confirmed to have terminated, the deregistration logic is triggered, removing the process identifier (Process ID, PID) and related process object references from the global security management list, ensuring that the list always reflects the set of currently active processes. This dynamic maintenance mechanism avoids the accumulation of invalid process handles in memory and effectively releases system resources.

[0026] Specifically, the atexit termination hook (on exit) can be configured. If the Python main process in the first child process layer exits normally, the `_cleanup_procs()` function registered by `atexit.register()` iterates through all Java child processes in the security management global list and terminates all active Java child processes. For example, in a Windows environment, the command `taskkill / F / T / PID` can be executed to forcibly terminate the specified process and its subtree; in a Linux or HarmonyOS environment, a SIGTERM (Signal Terminate) or SIGKILL (Signal Kill) signal is sent. After successfully terminating all processes, the security management global list is cleared or removed item by item, thus achieving thorough resource reclamation at the end of the main process's lifecycle. Furthermore, the `_CLEANUP_DONE` global flag can be added to the Java child processes in the security management global list to prevent duplicate execution; if the corresponding child process has already been cleaned up, the `_CLEANUP_DONE` global flag is added.

[0027] Signal handlers can also be configured in the global list of security management, such as registering SIGTERM and SIGINT signal handlers in the global list of security management. When the user enters the Ctrl+C (copy operation) interrupt command or the system generates a termination signal, the process cleanup operation is triggered, the default signal behavior is restored and the signal is resent to maintain the normal exit semantics of the system and avoid the system from suddenly crashing.

[0028] In scenarios involving frequent starting and stopping of screen mirroring connections, there might be situations where old screen mirroring bridging processes haven't been completely cleared before new ones have already started. To differentiate between processes launched in different batches, the concept of a generation is introduced. A globally incrementing counter is maintained in the main process at the first child process layer as the generation counter. Each time a new screen mirroring bridging process is attempted to start, this counter's value is incremented by one. For example, a generation counter can be configured, specifically an incrementing `self._stream_gen` counter maintained by the `ScreenCapture` class (screen capture class). Each call to `start_java_stream()` will increment this counter, meaning that each time a Java child process is started, the counter's value is incremented by one.

[0029] For each running process, a counter value is captured and saved at startup as the process's current generation. When a cleanup or maintenance operation needs to be performed on a process, the current generation value recorded by that process is read and compared with the counter generation value currently maintained in the main process. If the two values ​​are equal, it means that the process belongs to the latest startup batch; if the values ​​are not equal, it means that the process belongs to an older startup batch, and its related references may be outdated.

[0030] For example, when the background `_stream_loop` thread starts, it captures the current generation (`my_gen`). When the thread exits and cleans up, it checks the value of a counter as the counter generation `self._stream_gen` and verifies that `self._stream_gen == my_gen`. Only when the two sets of generation values ​​match is the `proc` reference cleared. This effectively prevents a common race condition: the cleanup thread of the old stream still holds a reference to `proc` (short for process, process object) after calling `_kill_proc_tree()`, while the new stream has already started and set a new `proc` reference—if the generations don't match, the old thread skips the clearing operation, and the `proc` of the new stream is preserved.

[0031] In asynchronous or multi-threaded environments, the thread responsible for cleaning up old processes may get blocked after obtaining the process object. During this blocking period, the main process may have already initiated a new round of screen mirroring connections, started a new process, and updated the generation counter. If the cleanup is performed directly based on the process reference obtained by the old thread, it is highly likely that the newly started process will be mistakenly killed, causing the screen mirroring service to be interrupted. By setting generation verification, the termination operation is only performed and the process is removed from the list when it is confirmed that the current generation of the process is inconsistent with the latest generation counter, that is, when it is confirmed that the process is indeed an expired process. Conversely, if the generation matches, even if the cleanup logic is delayed, the termination operation is not performed, thus protecting the survival of newly started processes. This mechanism strongly binds the lifecycle of a process to its generation identifier, ensuring that the system's state management remains accurate and robust even in complex scenarios where process startup and destruction occur concurrently.

[0032] The global security management list can also perform pre-boot remnant cleanup and context management. Pre-boot remnant cleanup specifically includes the following steps before each Java StreamBridge startup: (a) using "HDC fport rm" to clear old port forwarding rules; (b) using "HDC shell" to execute pidof (Process ID of) + kill (operating system signal command, process termination command) to terminate the residual screen_casting process on the user terminal side and delete the old scrcpy server library file; (c) using wmic (Windows) or pgrep (POSIX) to find and kill all residual StreamBridge Java processes associated with the current device serial number. Context management includes HOSDevice (HarmonyOS Device) implementing the __enter__ / __exit__ protocol, supporting the syntax "with HOSDevice.connect(sn) asdev: ...". The `__exit__` function calls the `stop()` method, which sequentially releases the screen stream (i.e., the video stream) and cleans up the touch, mouse, and keyboard controllers, ensuring that all resources are properly reclaimed. The `__del__` terminator also calls the `stop()` function to provide double protection.

[0033] S140. Analyze the current operating environment according to the preset environment level analysis rules to obtain the corresponding level analysis information.

[0034] After establishing the process management mechanism, the system needs to evaluate the current operating environment to determine the most suitable video processing and interaction strategy. This evaluation process is based on preset environment level resolution rules, ultimately outputting corresponding level resolution information. Environment level resolution rules define the standards for detecting the performance and compatibility of the operating environment, such as checking whether specific preset parsing libraries are installed and whether the user terminal's service operation status is normal. The level resolution information characterizes the performance level of the current environment, such as high-performance level, compatibility level, or low-performance level. For example, the system can determine the environment level by detecting the presence of dynamic link libraries for video decoding in the operating environment; it can also determine the environment level by sending a status probe request to the user terminal and determining the environment level based on the returned response data. Through environment level resolution, the system can intelligently match the optimal screen projection control strategy.

[0035] In a specific embodiment, step S140 includes the following sub-steps: determining whether the running screen projection bridging process is running normally; if the running screen projection bridging process is not running normally, determining the corresponding level resolution information as low performance level; if the running screen projection bridging process is running normally, sending a status acquisition request to the user terminal to obtain the service running status fed back by the user terminal; determining whether the current running environment has a preset resolution library installed and whether the service running status matches the preset status set in the environment level resolution rules; if the current running environment has a preset resolution library installed and the service running status matches the preset status, determining the corresponding level resolution information as high performance level; if the current running environment does not have a preset resolution library installed or the service running status matches the preset status, determining the corresponding level resolution information as compatibility performance level.

[0036] First, the lifecycle status of the screen mirroring bridging process is checked. The screen mirroring bridging process is a key component for achieving low-latency video stream forwarding, and its normal operation is a prerequisite for entering high-performance or compatibility modes. The process status is determined by querying whether the process identifier (PID) exists or by verifying whether the process responds to control commands issued by the system through a heartbeat detection mechanism. If the process is detected to have exited, is frozen, or is unresponsive, it is determined that it is not running normally.

[0037] When the screen mirroring bridging process becomes unavailable, the current operating environment is classified as low-performance. This indicates that the current environment cannot support real-time video streaming based on Java processes, and the system will automatically fall back to basic operating mode. In low-performance mode, subsequent video acquisition uses a screenshot polling mechanism within a pure Python environment, i.e., it obtains the display screen by calling the system screenshot command and retrieving the file. Although this method has lower real-time performance, it ensures that basic screen mirroring display functionality can still be achieved in constrained environments lacking high-performance operating conditions, thus guaranteeing the minimum availability of the system.

[0038] After confirming that the screen mirroring bridging process is active, a status retrieval request is sent to the user terminal. This request is used to probe whether the background services that the user terminal depends on are ready. For example, a query command is sent to the user terminal via the HarmonyOS screen mirroring development kit to inquire whether the service process responsible for screen capture (such as the screen copy library file) has started and is ready to transmit the video stream. After receiving the request, the user terminal reports its current service running status, such as whether the service is ready, the service is initializing, or the service has failed to start. Obtaining this service running status aims to rule out the risk of high-performance path failure due to service anomalies on the user terminal side.

[0039] A second, more refined verification is performed on the local operating environment and the service status on the user terminal side. The default parsing library is a necessary third-party library for decoding specific video stream formats (such as H.264 raw streams), such as the PyAV (Python binding for FFmpeg, a Python audio and video library based on FFmpeg) parsing library, used to provide efficient decoding capabilities. The default states set in the environment level resolution rules define the standard conditions for normal service operation, such as service startup or stream transmission channel establishment. Two checks are performed in parallel: first, checking whether the default parsing library has been successfully installed and loaded in the local operating environment; and second, verifying whether the service operating status reported by the user terminal meets the conditions defined by the default states. These two conditions together constitute a hard threshold for entering the high-performance level, ensuring that both the hardware and software environment are fully ready before enabling the high-performance decoding channel.

[0040] When the preset parsing library is installed and the service operation status fully meets the preset status, the parsing level is determined to be high-performance. At this time, the system has all the conditions to run the optimal video stream processing mode, namely, a high-performance decoding library is available locally, the service on the user terminal side is running stably, and the intermediate screen casting bridging process is communicating normally. Based on this high-performance level determination, the subsequent process configures the screen casting bridging process to output H.264 raw video stream data and uses the preset parsing library for low-latency decoding, thereby achieving a high frame rate and low latency real-time screen casting display experience, meeting the needs of interactive application scenarios with stringent real-time requirements.

[0041] When the preset parsing library is missing, or when the service operation does not meet the preset requirements even though the preset parsing library is installed (i.e., at least one of the above two conditions is not met), the parsing information is determined to be a compatibility performance level. The compatibility performance level represents a compromise operating state: while it cannot achieve ultimate H.264 raw stream decoding performance, the system can still utilize the screen casting bridging process for video stream transmission. At this level, subsequent process control of the screen casting bridging process pre-decodes and encodes the video stream into a common JPEG format image for transmission. Local display only needs to use the basic image library, without relying on the preset parsing library. This approach, at the cost of sacrificing some decoding efficiency, expands the compatibility range with different software environments and device states, ensuring that even in non-ideal environments, it still provides a better screen casting effect than the lower performance level.

[0042] S150. If the grade parsing information is not a low performance grade, the video stream corresponding to the screen casting file is obtained through the first sub-process layer and the screen casting bridging process and then displayed.

[0043] If the environment level analysis information obtained from the above steps indicates that the current environment is not at a low performance level (i.e., it belongs to a high performance or compatibility performance level), then video stream analysis is used for screen casting. In this mode, the first sub-process layer and the screen casting bridging process work together to parse the screen casting file and generate a video stream. The non-low performance level corresponds to a high-performance decoding mode and a compatibility encoding mode. The screen casting bridging process is responsible for extracting the raw video data from the screen casting file and selecting different processing paths based on the environment level. For example, when the environment supports high-performance decoding, the screen casting bridging process outputs the raw video unit sequence, which is scanned and decoded by the scanning process running in the first sub-process layer, directly obtaining the video stream for display. This process avoids intermediate encoding and decoding stages, achieving low-latency transmission. When the environment only supports compatibility performance, the screen casting bridging process encodes the raw video unit sequence into image data packets, and the first sub-process layer extracts multiple frames of images from the data packets through an image extraction thread to form a video stream for display. Through this differentiated analysis method, the system can provide a smooth screen casting experience under different performance environments.

[0044] In a specific embodiment, step S150 includes the following sub-steps: outputting the original video unit sequence corresponding to the projected file through the output thread in the projection bridging process; if the level resolution information is a high-performance level, scanning and decoding the original video unit sequence through the scanning process running in the first sub-process layer to obtain a video stream and display it; if the level resolution information is a compatibility performance level, encoding the original video unit sequence through the encoding thread in the projection bridging process to output the corresponding image data packet; and extracting the corresponding multi-frame images from the image data packet through the image extraction thread running in the first sub-process layer to form a video stream and display it.

[0045] After the screen casting bridging process establishes a connection with the device and successfully captures screen data, the output thread within the screen casting bridging process is responsible for parsing the acquired raw screen casting file to obtain the corresponding raw video unit sequence. This raw video unit sequence is not a simple collection of image data, but a structured byte stream sequence representing the basic coding units of the video stream. In terms of data source, it is obtained in real-time by the screen casting bridging process through calling the underlying interface provided by the HarmonyOS screen casting development kit, utilizing the HarmonyOS Screen Casting service. In terms of generation, this sequence is generated through the protocol encapsulation logic within the screen casting bridging process, specifically using a binary protocol with a 4-byte big-endian length prefix and a data body. This sequence specifically contains a series of video coding data, such as, but not limited to, H.264 encoded NAL (Network Abstraction Layer) units, which carry complete intra-frame prediction, inter-frame prediction, and residual data information. The aforementioned original video unit sequence serves as the sole data source for subsequent video stream parsing, ensuring consistent underlying data input regardless of whether it is in high-performance mode or compatibility mode, thus realizing a complete data flow link from underlying device acquisition to inter-process transmission.

[0046] When the system determines that the current operating environment's resolution information is at a high-performance level according to the method described in the above embodiments, it means that the host environment where the first subprocess layer resides has the necessary decoding capabilities, such as having a preset parsing library (e.g., PyAV library) installed and sufficient hardware resources. At this time, an efficient direct decoding path is adopted. The first subprocess layer starts and runs a scanning process, which is responsible for extracting video frames from the original video unit sequence. Specifically, the scanning process first reads the byte stream data output by the projection bridging process through the input pipe, and then executes the scanning logic, that is, searching for specific start codes (e.g., 0x00000001 or 0x000001) in the byte stream to locate and segment independent NAL (Network Abstraction Layer) units. After completing the unit segmentation, the scanning process calls the decoding interface of the preset parsing library (e.g., av.CodecContext.parse() and av.CodecContext.decode()) to decode the compressed H.264 data into uncompressed video frame data (e.g., YUV or RGB format AVFrame objects). Finally, the decoded video frame data is converted into an image format recognizable by the display engine (e.g., converted to a PIL Image object via frame.to_image()) and rendered to obtain the video frame. This approach bypasses the encoding and decoding of intermediate image formats such as JPEG, directly generating the display image from the raw video data, thereby minimizing data processing latency and CPU computational overhead. The frame rate can reach 60fps, making it suitable for real-time interactive control and computer vision (CV) analysis scenarios.

[0047] When the performance rating is determined to be a compatibility level, it indicates that although the current environment can support the normal operation of the screen mirroring bridging process, it may be unable to directly execute complex raw video decoding operations at the first subprocess layer due to the lack of a specific high-performance parsing library (such as PyAV) or other environmental limitations. To ensure the availability of the screen mirroring function, this application shifts the computational load of video processing to the screen mirroring bridging process with a Java runtime environment. Under this path, the encoding thread in the screen mirroring bridging process is activated to process the raw video unit sequence obtained in the above steps. The encoding thread first decodes the raw H.264NAL units to restore the raw image frames, and then uses the built-in encoder to re-encode the image frames into a common image compression format, such as JPEG. After encoding, the encoding thread encapsulates the encoded image data according to a preset communication protocol and outputs the corresponding image data packet. This image data packet is encapsulated using a length prefix protocol similar to that of the raw video unit sequence, but its data body content is a single-frame image file that has been encoded (such as complete JPEG file data). By completing the conversion from raw video to a common image format within the screen casting bridging process, the first sub-process layer only needs to perform lightweight data reading and simple image extraction, thereby reducing the stringent requirements on the dependent libraries of the first sub-process layer and achieving functional compatibility in constrained environments.

[0048] At the compatibility performance level, the image data packets generated through the above steps are transmitted to the first sub-process layer. To maintain the continuity and real-time performance of the video stream, the first sub-process layer runs a dedicated image extraction thread. This image extraction thread is responsible for continuously listening to and reading image data packets from the projection bridging process. In terms of data acquisition, the image extraction thread determines the boundary of each image packet by parsing the 4-byte length prefix in the data packet, thereby accurately extracting complete single-frame image data from the continuous byte stream. Subsequently, the image extraction thread uses basic image processing libraries (such as the Python standard library or the lightweight PIL library) to decode the extracted image data, restoring it into displayable image objects. This process is executed cyclically, converting the discrete image data packets received in chronological order into a sequence of continuous multi-frame images, thus forming a video stream. Finally, these multi-frame images are refreshed to the display interface sequentially according to the receiving time sequence, forming a dynamic video image. By running the image extraction thread independently, the main thread is avoided from being blocked due to network jitter or decoding time, ensuring smooth display at the compatibility performance level.

[0049] The aforementioned steps work together to construct a video stream processing system that adapts to the capabilities of the operating environment. The raw video unit sequence output by the projection bridging process serves as a unified data source and is dynamically allocated to different processing chains based on the environment level parsing information: in a high-performance environment, low-latency direct decoding is achieved using the scanning process of the first sub-process layer; in a compatible environment, format adaptation is performed through the encoding thread of the projection bridging process, and then display is completed via the image extraction thread of the first sub-process layer. This design allows this application to fully utilize the hardware and software resources of high-performance environments to achieve low-latency projection, while ensuring functional availability in resource-constrained or dependency-deficient environments, thus solving the technical problem that a single video processing mode cannot adapt to diverse operating environments.

[0050] S160. If the level parsing information is a low performance level, start a screenshot process in the first sub-process layer to capture the corresponding display screen from the screen projection file for display.

[0051] If the performance level analysis indicates that the current operating environment is at a low performance level, meaning a stable video stream transmission channel cannot be established, the system automatically downgrades to screenshot polling mode. A screenshot process is started in the first sub-process layer, responsible for periodically capturing the corresponding display screen from the projection file. The screenshot process performs the capture operation according to the configured screenshot frequency and generates a unique temporary filename for each display screen to avoid concurrent read / write conflicts. For example, the screenshot process can call the system-provided screen capture interface (such as the screencap command) to obtain the current screen image, or perform pixel-level reading of the projection file buffer through a graphics processing library. The screenshot process displays the captured display screen on the in-vehicle terminal screen in real time. Although the real-time performance of the screenshot polling mode is lower than that of the video stream mode, it ensures the availability of the projection function and meets basic testing requirements in low-performance environments, achieving automatic fallback protection for multimodal video streams.

[0052] In a specific embodiment, step S160 includes the following sub-steps: the screenshot process captures the corresponding display screen from the projection file according to the configured screenshot frequency and records the corresponding image capture result; each display screen generates a unique matching temporary file name; if the image capture result is successful, the display screen is displayed; if the image capture result is unsuccessful, it is determined whether the currently recorded cumulative number of failures exceeds the number threshold; if it does not exceed the number threshold, the screenshot frequency is adjusted according to a preset backoff algorithm, and then the step of capturing the corresponding display screen according to the configured screenshot frequency is returned to execution; if it exceeds the number threshold, the operation of capturing the display screen through the screenshot process is terminated.

[0053] In low-performance mode, the screenshot process is the sole means of acquiring video data, and its operational stability directly determines the user experience. The screenshot process is configured to periodically perform screenshot actions according to a preset screenshot frequency, which defines the time interval between two consecutive screenshot operations. Considering the balance between smooth image updates and system load, the initial value of the screenshot frequency can be set to 500 milliseconds (i.e., 2 frames per second). When performing a screenshot operation, the screenshot process calls the underlying interface provided by the first child process layer (such as the HDC shell snapshot_display command) to initiate a screenshot request to the user terminal. After each screenshot action is completed, the screenshot process records the corresponding image capture result, which represents the success or failure of the operation. The criteria for judgment include, but are not limited to: whether the command return code is 0, whether the size of the received data packet meets expectations, and whether the file checksum is correct. By recording the image capture result, the system constructs the basic feedback signal for closed-loop screenshot control.

[0054] To support multi-threaded concurrent screenshots and avoid file overwriting conflicts, the screenshot process executes unique temporary filename generation logic when saving image data for each displayed frame. This temporary filename generation process employs a Universally Unique Identifier (UUID) algorithm or a high-precision random number combination strategy based on timestamps to ensure uniqueness across the entire system and over long time spans. For example, the generated temporary filename can include a timestamp, a random string, and a process identifier, such as screenshot_20231027_153045_a1b2c3d4.jpg. This temporary filename serves as a unique identifier for the file storage path, allowing image files written by multiple screenshot tasks or multiple retries to coexist on the storage medium without interference, even in high-frequency concurrent screenshot scenarios. This naming mechanism avoids data loss or reading errors caused by duplicate filenames, providing a reliable index for subsequent image display and cleanup operations.

[0055] When the recorded image capture result indicates a successful state, it means that the image data of the corresponding display screen has been completely transmitted from the user's terminal to the local machine and saved as an image file. At this time, the screenshot process reads the image data corresponding to the temporary file name through the image loading interface and renders it to the display interface. The display process includes decoding the image file into bitmap data, adjusting the resolution to fit the display window, and performing a screen refresh operation, so that the user can see the screen projection content presented by the vehicle terminal. If there are consecutive successful screenshots, the system will present a series of static display screens in a continuous manner according to the screenshot frequency, forming a dynamic video playback effect under the effect of human visual persistence. After the image is loaded and displayed, the file corresponding to the temporary file name can be marked as cleanable so that the subsequent garbage collection mechanism can release storage space.

[0056] If the recorded image capture result indicates a failure, such as a communication timeout, abnormal device response, or storage write error preventing the capture operation from producing a valid image file, the system triggers an exception handling procedure. This procedure first reads the currently recorded cumulative failure count, a counter used to track the total number of failed capture operations within a continuous or specific time window. The system compares the read cumulative failure count with a preset threshold, which is the upper limit set by the system for the number of consecutive retries or allowed failures, for example, it can be set to 10 times. This judgment logic constitutes the precondition for the system's circuit breaker protection mechanism, aiming to distinguish between transient faults and persistent unavailability, preventing the system from getting stuck in an invalid infinite retry loop when the device is offline or the network is completely interrupted.

[0057] When the cumulative number of failures has not exceeded the threshold, the system assumes that the current screenshot failure may be caused by momentary network fluctuations or temporary device overload, and there is still a possibility of successful retry. At this point, the system activates a dynamic adjustment mechanism, adjusting the screenshot frequency according to a preset backoff algorithm. The backoff algorithm is configured to exponentially increase the screenshot interval as the number of failures increases. For example, after the first failure, the screenshot frequency is adjusted, such as to 1 (waiting 1 second to retry), 0.5 (waiting 2 seconds to retry), 0.25 (waiting 4 seconds to retry), and so on, until a preset maximum waiting limit (e.g., 30 seconds) is reached. This exponential backoff strategy effectively reduces the impact of high-frequency screenshot requests on the vehicle terminal system resources when the device is busy, allowing sufficient recovery time for the device. After calculating the new waiting time and updating the screenshot frequency according to the backoff algorithm, the screenshot process returns to the above steps, attempting to capture the corresponding display screen again based on the adjusted screenshot frequency, forming an adaptive closed loop of failure → backoff waiting → retry.

[0058] If the cumulative number of failures reaches or exceeds the threshold, the system determines that the current environment or device status cannot meet the normal screenshot display requirements. Continuing to retry not only wastes computing resources but may also block the main thread. At this point, the system triggers a circuit breaker mechanism, forcibly terminating the operation of capturing and displaying the screen through the screenshot process. The termination operation includes stopping the screenshot timer, exiting the screenshot loop thread, and releasing related communication connection resources. After termination, the system can provide feedback to the user or upper-layer applications indicating that the screen mirroring connection is interrupted or the display service is unavailable. This forced termination measure ensures the robustness of the overall system operation, avoids system freezes or resource exhaustion caused by infinite retries in low-performance mode, and also provides users with a clear stop signal for troubleshooting network connectivity or device status.

[0059] Through the above steps, the screenshot process has built a complete lifecycle management mechanism at low performance levels, including frequency control, concurrency safety, successful display, failure counting, backoff retry, and over-limit circuit breaking, to ensure that stable and reliable screen projection display services can still be provided in environments lacking high-performance video stream support.

[0060] S170. If a touch command is received from the user terminal, the command execution mode corresponding to the level parsing information is obtained according to the preset mode selection rules.

[0061] During screen projection, if the vehicle terminal receives a touch command from the user terminal, it needs to select an appropriate command execution mode based on the current environment. This selection process follows preset mode selection rules, which establish a mapping relationship between level parsing information and command execution modes. For example, the mode selection rules configure a basic protocol mode mapped to level parameters and an asynchronous queue mode not mapped to level parameters. The level parameters include two levels: high-performance level and compatibility performance level. By matching the level parsing information with the level parameters, it can be determined whether the level parsing information matches the level parameters, thus obtaining the command execution mode corresponding to the matching result according to the mapping relationship configured in the mode selection rules. The command execution mode defines the specific way touch commands are parsed and executed; different modes correspond to different performance and compatibility characteristics. A configurable three-layer touch injection architecture is supported, with command execution modes including basic protocol mode, asynchronous queue mode, and independent command mode. For example, if the resolution information indicates a high-performance or compatibility level, the mode selection rule can match the basic protocol mode, using the input pipe of a subprocess to send the touch protocol text line, achieving low-latency response. If the resolution information indicates a low-performance level, the mode selection rule can match the asynchronous queue mode, placing the touch command in a thread-safe queue for execution by a background thread, avoiding blocking the main thread. Through the mode selection rule, the system can ensure that the touch operation matches the current projected video stream or displayed screen.

[0062] In a specific embodiment, step S170 includes the following sub-steps: determining whether the level parsing information matches the level parameter set in the mode selection rule; if the level parsing information matches the level parameter, determining the corresponding instruction execution mode as the basic protocol mode; if the level parsing information does not match the level parameter, determining the corresponding instruction execution mode as the asynchronous queue mode.

[0063] The level parsing information obtained in the preceding steps is used as input and matched against the level parameters defined in the preset mode selection rules. The level parameters are the core criteria in the mode selection rules, used to determine whether the operating environment has the capability to handle high-speed, high-frequency touch commands. During system initialization or configuration, historical performance test data is analyzed to identify operating environment identifiers with high real-time response capabilities, and these identifiers or their combinations are solidified into level parameters. For example, if historical performance test data analysis determines that the operating environment of the test process meets the high-performance level or compatibility performance level and can stably execute touch commands through the basic protocol mode (execution failure rate less than 0.5%), then the configured level parameters include both the high-performance level and the compatibility performance level; if historical performance test data analysis determines that the operating environment of the test process meets the high-performance level or can stably execute touch commands through the basic protocol mode (execution failure rate less than 0.5%), then the configured level parameters only include the high-performance level.

[0064] For example, if the grade parsing information determined in the aforementioned steps is a high-performance grade or a compatibility performance grade, and the grade parameter set in the mode selection rule includes both of these grades, then the matching result is determined to be yes. Conversely, if the grade parsing information is a low-performance grade, then the matching result is determined to be no. This judgment logic provides a decision-making basis for subsequently selecting a specific touch command transmission channel.

[0065] When the level resolution information corresponding to the current running environment matches the level parameters, the instruction execution mode is established as the basic protocol mode. The basic protocol mode is a low-latency touch injection mechanism based on the standard input stream of a Java subprocess. In this mode, touch instructions no longer go through the scheduling overhead of command-line tools, but are directly sent via the stdin (Standard Input) pipe of the Java subprocess. The `_write()` method is used to write instruction information such as "D:x:y" into the stdin pipe and immediately pushes it for execution. In this mode, the movement events of touch instructions have a built-in rate limiting mechanism: a maximum of 20 times / second (50ms interval), and skipping tiny displacements of less than 10 pixels in both the horizontal and vertical directions compared to the last sent position, effectively avoiding stdin pipe congestion during drag operations. This mechanism utilizes the Socket connection channel established between the JavaStreamBridge process and the HarmonyOS device to achieve millisecond-level forwarding of touch instructions. When the stdin pipe is closed, it is automatically marked as "dead," and subsequent calls silently skip it.

[0066] At the execution level, touch coordinate data is rapidly pushed to the pipeline buffer by calling the write interface, and a refresh operation is immediately performed to ensure that the command takes effect instantly. By adopting the basic protocol pattern, the high-performance characteristics of the Java environment are fully utilized, eliminating the process creation and destruction overhead of command-line calls, significantly reducing the end-to-end latency of touch operation from receiving the response, and meeting the needs of interactive scenarios with high real-time requirements.

[0067] When it is determined that the level resolution information corresponding to the current operating environment does not match the level parameters (e.g., in low-performance scenarios such as screenshot polling mode), the instruction execution mode is established as asynchronous queue mode. Asynchronous queue mode is a touch processing architecture based on a producer-consumer model, designed to solve the problem of calling threads being blocked due to synchronously waiting for system responses in low-performance environments. In this mode, touch instructions are not executed immediately, but are encapsulated into task objects containing execution logic and parameters.

[0068] Specifically, in asynchronous queue mode, touch commands are wrapped and placed into a thread-safe queue (closure function + parameters) in the Python standard library's `queue.Queue`, which is then consumed and executed one by one by a background daemon thread. The caller returns immediately after calling the touch methods Down, Move, Up, Click, and Swipe, without being blocked by the HDC child process. This mode is used in screenshot polling mode (corresponding to the lower performance level mentioned above) to ensure smooth response of the calling thread. The `stop()` method emptys the remaining commands in the queue and waits for the worker thread to finish.

[0069] By introducing an asynchronous queue pattern, time-consuming I / O operations are decoupled from the main business thread, ensuring that the execution flow of the main program will not be suspended even if the underlying processing speed is slow, thereby guaranteeing the smoothness and stability of automated test scripts or control logic.

[0070] In a specific embodiment, before determining the corresponding instruction execution mode as the basic protocol mode, the method further includes: determining whether the current operating environment is a test environment; if it is a test environment, determining the corresponding instruction execution mode as the independent command mode; if it is not a test environment, determining the corresponding instruction execution mode as the basic protocol mode.

[0071] Before determining the execution mode of touch commands, the attributes of the current operating system environment are first identified. The type of the current operating environment directly determines the different requirements of touch operations for debuggability, execution isolation, and response latency. Environment type determination is achieved by reading system configuration identifiers, environment variables, or detecting the status of specific debug ports. For example, it checks for the presence of specific test environment flags (such as the preset environment variable TEST_MODE=true), whether the currently executed script namespace belongs to a test framework (such as pytest, unittest), or whether the in-vehicle terminal is currently in developer debug mode, among other comprehensive conditions. If any of the above test characteristics are detected, the current operating environment is determined to be a test environment. This environment identification mechanism aims to provide the most suitable interaction strategy for different operating scenarios, ensuring the accuracy of the testing process and the stability of the production environment.

[0072] When the current runtime environment is determined to be a test environment, the instruction execution mode is set to independent command mode. Independent command mode differs from the basic protocol mode based on Java stdin pipes. Its technical essence lies in constructing and executing independent command-line calls for each touch operation (including Down, Move, Up, Click, Long Press, and Swipe). In this mode, touch instructions no longer reuse the high-speed data pipeline of persistent connections. Instead, they generate independent processes to execute operation instructions by calling a first child process layer compatible with the user terminal system program.

[0073] For example, each touch command (press / release / move / click / long press / swipe) executes an independent "hdcshell uinput -M" command. It supports specifying touch point numbers (e.g., touch points 0 to 9) via the -c parameter, allowing precise control over the specific location of touch events. For swipe operations, this mode automatically generates several intermediate movement events between the start and end points using an interpolation algorithm, sending them sequentially at set intervals to simulate a realistic and smooth swipe trajectory. Since each operation corresponds to an independent system call, the independent command mode offers the strongest compatibility and traceability, facilitating step-by-step debugging, error localization, and logging during the testing phase. It effectively avoids test misjudgments or unreproducible behavior issues caused by abnormal underlying communication pipe states (such as Java process blocking or buffer accumulation). This mode has the best compatibility, is applicable to all HarmonyOS versions, and is suitable for automated test script scenarios.

[0074] If the current operating environment is not identified as a test environment, i.e., it is currently in a production environment or a regular operating environment, then the instruction execution mode will be determined as the basic protocol mode.

[0075] S180. Parse the touch instruction according to the instruction execution mode and respond to the touch operation corresponding to the touch instruction in the video stream or the display screen.

[0076] After determining the instruction execution mode, the system parses the touch instructions according to the mode and responds to the corresponding touch operation in the currently displayed video stream or screen. Touch operations include actions such as pressing, moving, and lifting the touch, which the system simulates by sending control signals to the user terminal. For example, in the basic protocol mode, the system converts the touch instructions into text lines conforming to the protocol specification, sends them to the screen projection bridging process through the input pipe of the first sub-process layer, and then passes them to the user terminal for execution. This mode has a built-in rate limiting mechanism to avoid pipe congestion. In the asynchronous queue mode, the system encapsulates the touch instructions into task objects and pushes them into a queue, where they are retrieved and executed by an independent worker thread, ensuring immediate response to the touch instructions. By accurately executing touch operations according to the instruction execution mode, this application achieves reliable interaction in different screen projection scenarios, effectively improving the accuracy of remote control and user experience.

[0077] The aforementioned technical solutions work together to address compatibility, performance, and security issues during the screen projection process between vehicle-mounted terminals and user terminals. The dual-layer architecture design of the first and second sub-process layers effectively separates basic command calls from high-performance video stream relay, ensuring system adaptability under different operating environments. The introduction of a global security management list, combined with process registration and cleanup mechanisms, effectively prevents process leaks and resource residues, enhancing system robustness. The coordination between environment level parsing rules and the multi-modal video stream system (including video stream parsing and screenshot rollback) enables the system to automatically select the optimal projection path based on current hardware and software conditions, ensuring low latency and high performance while maintaining functional availability in constrained environments. Furthermore, the combination of mode selection rules and a multi-layered touch injection system allows for precise matching of touch command execution strategies with the projection mode, achieving stable and reliable touch operation whether using the basic protocol mode transmitted through pipelines or the asynchronous mode using queue buffering. The overall technical solution, through adaptive environment awareness and hierarchical processing mechanisms, constructs a highly robust, low-latency, and easily maintainable screen projection control system.

[0078] This invention discloses a screen projection control method based on process lifecycle security management, relating to the field of vehicle-mounted terminal screen projection control technology. The method includes: configuring a first sub-process layer compatible with the user terminal system program, and starting a second sub-process layer that runs a screen projection bridging process; adding the process of the second sub-process layer to the global security management list; parsing the current operating environment to obtain level resolution information; depending on whether the level resolution information is non-low performance or low performance, parsing the video stream through the screen projection bridging process or starting a screenshot process to capture and display the screen; obtaining the instruction execution mode corresponding to the level resolution information according to mode selection rules, and responding to touch operations. This application achieves system compatibility and smooth screen projection by using a first sub-process layer compatible with the user terminal system program and a second sub-process layer running the screen projection bridging process, and improves the stability of screen projection control through process lifecycle security management.

[0079] This invention also provides a screen projection control device based on process lifecycle security management. This device can be configured in an in-vehicle terminal and is used to execute any embodiment of the aforementioned screen projection control method based on process lifecycle security management. Specifically, please refer to... Figure 3 , Figure 3 This is a schematic block diagram of a screen projection control device based on process lifecycle security management provided in an embodiment of the present invention.

[0080] like Figure 3As shown, the screen projection control device 100 based on process lifecycle security management includes a first sub-process layer configuration unit 110, a process running unit 120, a process adding unit 130, a level parsing information acquisition unit 140, a video stream display unit 150, a display screen display unit 160, an instruction execution mode acquisition unit 170, and a touch operation response unit 180.

[0081] The first subprocess layer configuration unit 110 is used to configure the first subprocess layer if a screen projection file is received from the user terminal. The first subprocess layer is compatible with the system program of the user terminal.

[0082] The process execution unit 120 is used to start the second sub-process layer through the first sub-process layer and run the screen projection bridging process in the second sub-process layer.

[0083] The process addition unit 130 is used to add all processes running in the second subprocess layer to a preset global security management list to achieve process security management.

[0084] The level resolution information acquisition unit 140 is used to parse the current operating environment according to the preset environment level resolution rules and obtain the corresponding level resolution information.

[0085] The video stream display unit 150 is used to parse and display the video stream corresponding to the projection file through the first sub-process layer and the projection bridging process if the level parsing information is not low performance level.

[0086] The display unit 160 is used to start a screenshot process in the first sub-process layer to capture the corresponding display screen from the projection file and display it if the level parsing information is low performance level.

[0087] The instruction execution mode acquisition unit 170 is used to acquire the instruction execution mode corresponding to the level parsing information according to a preset mode selection rule if a touch instruction is received from the user terminal.

[0088] The touch operation response unit 180 is used to parse the touch instruction according to the instruction execution mode and respond to the touch operation corresponding to the touch instruction in the video stream or the display screen.

[0089] The screen projection control device based on process lifecycle security management provided in this embodiment of the invention applies the aforementioned screen projection control method based on process lifecycle security management, relating to the field of vehicle terminal screen projection control technology. The method includes: configuring a first sub-process layer compatible with the user terminal system program, and starting a second sub-process layer that runs a screen projection bridging process through it; adding the process of the second sub-process layer to the global security management list; parsing the current operating environment to obtain level resolution information; depending on whether the level resolution information is a non-low performance level or a low performance level, parsing the video stream through the screen projection bridging process or starting a screenshot process to capture and display the screen; obtaining the instruction execution mode corresponding to the level resolution information according to the mode selection rules, and responding to touch operations. This application achieves system compatibility and smooth screen projection by using a first sub-process layer compatible with the user terminal system program and a second sub-process layer running the screen projection bridging process, and improves the stability of screen projection control through process lifecycle security management.

[0090] The aforementioned screen projection control device based on process lifecycle security management can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0091] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device can be a vehicle-mounted terminal used to execute a screen projection control method based on process lifecycle security management to achieve screen projection connection with a user terminal.

[0092] See Figure 4 The computer device 500 includes a processor 502, a memory, and a communication interface 505 connected via a communication bus 501. The memory may include a storage medium 503 and internal memory 504.

[0093] The storage medium 503 can store the operating system 5031 and the computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute a screen projection control method based on process lifecycle security management. The storage medium 503 can be a volatile storage medium or a non-volatile storage medium.

[0094] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0095] The internal memory 504 provides an environment for the computer program 5032 in the storage medium 503 to run. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a screen projection control method based on process lifecycle security management.

[0096] This communication interface 505 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0097] The processor 502 is used to run the computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned screen projection control method based on process lifecycle security management.

[0098] Those skilled in the art will understand that Figure 4 The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 4 The embodiments shown are consistent and will not be described again here.

[0099] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0100] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium stores a computer program, wherein when executed by a processor, the computer program implements the steps included in the screen projection control method based on process lifecycle security management described above.

[0101] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0102] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A screen projection control method based on process lifecycle security management, characterized in that, The method is applied in an in-vehicle terminal, which establishes a communication connection with a user terminal to transmit data information. The method includes: If a screen-projected file is received from the user terminal, a first sub-process layer is configured, which is compatible with the system program of the user terminal. The second subprocess layer is started through the first subprocess layer, and the screen casting bridging process is run in the second subprocess layer; All processes running in the second child process layer are added to a pre-defined global security management list to achieve process security management; The current operating environment is analyzed according to the preset environment level analysis rules to obtain the corresponding level analysis information; If the grade resolution information is not a low performance grade, the video stream corresponding to the screen casting file is obtained through the first sub-process layer and the screen casting bridging process and then displayed. If the performance level is low, a screenshot process is started in the first sub-process layer to capture the corresponding display screen from the projection file for display. If a touch command is received from the user terminal, the command execution mode corresponding to the level parsing information is obtained according to the preset mode selection rules; The touch instruction is parsed according to the instruction execution mode and the touch operation corresponding to the touch instruction is responded to in the video stream or the display screen.

2. The screen projection control method based on process lifecycle security management according to claim 1, characterized in that, The step of adding all processes running in the second sub-process layer to a preset global security management list to achieve process security management includes: If a process in the global security management list terminates, the terminated process will be removed from the global security management list. If the main process in the first child process layer exits normally, then all processes in the global security management list are terminated and removed. Determine whether the current generation of the process in the global security management list matches the counter generation; If the current generation of the process does not match the counter generation, the process is terminated and removed from the global security management list.

3. The screen projection control method based on process lifecycle security management according to claim 1 or 2, characterized in that, The process of parsing the current operating environment according to preset environment level resolution rules to obtain corresponding level resolution information includes: Determine whether the screen projection bridging process is running normally; If the screen mirroring bridging process is not running normally, the corresponding performance level is determined to be low. If the screen mirroring bridging process is running normally, a status acquisition request is sent to the user terminal to obtain the service running status feedback from the user terminal. Determine whether the current operating environment has the preset parsing library installed and whether the service running status matches the preset status set in the environment level parsing rules; If the current operating environment has a preset parsing library installed and the service operating status matches the preset status, the corresponding parsing information is determined to be a high-performance level. If the current operating environment does not have the preset parsing library installed, or if the service operating status matches the preset status, the corresponding parsing information is determined to be the compatibility performance level.

4. The screen projection control method based on process lifecycle security management according to claim 3, characterized in that, The step of parsing and displaying the video stream corresponding to the screen-casting file through the first sub-process layer and the screen-casting bridging process includes: The output thread in the screen casting bridging process outputs the original video unit sequence corresponding to the screen casting file. If the grade parsing information is a high-performance grade, the original video unit sequence is scanned and decoded by the scanning process running in the first sub-process layer to obtain a video stream and display it; If the level resolution information is a compatibility performance level, the original video unit sequence is encoded by the encoding thread in the screen projection bridging process, and the corresponding image data packet is output. The image extraction thread running in the first subprocess layer extracts the corresponding multi-frame images from the image data packet, assembles them into a video stream, and displays it.

5. The screen projection control method based on process lifecycle security management according to claim 4, characterized in that, The step of starting a screenshot process in the first sub-process layer to capture the corresponding display screen from the projection file for display includes: The screenshot process captures the corresponding display screen from the screen projection file according to the configured screenshot frequency and records the corresponding image capture results; a unique matching temporary file name is generated for each display screen; If the image capture result is successful, the display screen will be shown. If the image capture result is a failure, determine whether the cumulative number of failures in the current record exceeds the threshold. If the number of times does not exceed the threshold, the screenshot frequency is adjusted according to the preset backoff algorithm, and then the process returns to the step of capturing the corresponding display screen according to the configured screenshot frequency. If the number of attempts exceeds the threshold, the operation of capturing and displaying the screen through the screenshot process will be terminated.

6. The screen projection control method based on process lifecycle security management according to claim 5, characterized in that, The step of obtaining the instruction execution mode corresponding to the level parsing information according to the preset mode selection rules includes: Determine whether the level parsing information matches the level parameters set in the mode selection rules; If the level parsing information matches the level parameter, the corresponding instruction execution mode is determined to be the basic protocol mode; If the level parsing information does not match the level parameter, the corresponding instruction execution mode is determined to be asynchronous queue mode.

7. The screen projection control method based on process lifecycle security management according to claim 6, characterized in that, Before determining the corresponding instruction execution mode as the basic protocol mode, the following steps are also included: Determine if the current runtime environment is a test environment; If it is a test environment, the corresponding instruction execution mode is determined to be the independent command mode; If it is not a test environment, determine the corresponding instruction execution mode as the basic protocol mode.

8. A screen projection control device based on process lifecycle security management, characterized in that, The device is configured in an in-vehicle terminal, which communicates with a user terminal to transmit data information. The device is used to execute the screen projection control method based on process lifecycle security management as described in any one of claims 1-7, and the device includes: The first subprocess layer configuration unit is used to configure the first subprocess layer if a screen projection file is received from the user terminal. The first subprocess layer is compatible with the system program of the user terminal. The process execution unit is used to start the second sub-process layer through the first sub-process layer and run the screen projection bridging process in the second sub-process layer. The process addition unit is used to add all processes running in the second sub-process layer to a preset global security management list to achieve process security management; The level resolution information acquisition unit is used to parse the current operating environment according to the preset environment level resolution rules and obtain the corresponding level resolution information; The video stream display unit is used to parse the video stream corresponding to the projection file through the first sub-process layer and the projection bridging process and display it if the level parsing information is not low performance level. The display screen unit is used to start a screenshot process in the first sub-process layer to capture the corresponding display screen from the projection file and display it if the level parsing information is a low performance level. The instruction execution mode acquisition unit is used to acquire the instruction execution mode corresponding to the level parsing information according to the preset mode selection rules if a touch instruction is received from the user terminal. A touch operation response unit is used to parse the touch instruction according to the instruction execution mode and respond to the touch operation corresponding to the touch instruction in the video stream or the display screen.

9. A computer device, characterized in that, The device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the screen projection control method based on process lifecycle security management as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the screen projection control method based on process lifecycle security management as described in any one of claims 1-7.

Images