Multi-signal-source integrated display method and system of detection vehicle, medium and product

Abstract

A multi-signal-source integrated display method, system, medium and product for a detection vehicle relate to the field of image display, and the method comprises the following steps: in response to access triggering of multiple signal sources, reading original heterogeneous data streams in parallel; obtaining the current driving speed and network bandwidth of the vehicle, determining image scaling and frame rate parameters corresponding to the driving speed and the network bandwidth, and obtaining a rendering control instruction; executing the rendering control instruction, and performing decoding and pixel resampling processing on the original heterogeneous data stream to generate an independent layer object; calling a preset screen layout coordinate file, mapping the plurality of independent layer objects to the same canvas coordinate system for pixel superposition, and generating a single-frame integrated display image; and writing the single-frame integrated display image into a video memory frame buffer area, and driving a display terminal to output the single-frame integrated display image. By implementing the application, the rendering quality of the display picture can be improved under the dynamic working condition of the vehicle.

Description

Technical Field

[0001] This application relates to the field of image display, and in particular to a method, system, medium, and product for integrated display of multiple signal sources on a detection vehicle. Background Technology

[0002] As a mobile platform for performing special missions, reconnaissance vehicles need to acquire and display multiple heterogeneous signals from drones, vehicle-mounted cameras, reconnaissance robots, and various environmental sensors in real time. To enable personnel inside the vehicle to simultaneously observe these multiple signals on a single display terminal, a technological means is needed to centrally process and display video and data streams from different sources.

[0003] In related technologies, multi-signal source integrated display technology typically uses a hardware splitter to connect each signal source to the display terminal. That is, each signal source device is connected to the input port of the hardware splitter through an independent physical transmission cable. The image processing chip inside the hardware splitter performs fixed-ratio scaling and splicing processing on the video signals received from each input port according to preset circuit logic, generating a composite video signal containing multiple sub-screens, which is then transmitted to the display terminal for display through a single output port.

[0004] However, during operation, reconnaissance vehicles are often in dynamic conditions with driving vibration or network signal fluctuations. When the network bandwidth decreases, causing the input signal frame rate to drop, or when the vehicle vibrates violently, affecting the stability of the physical connection, the hardware screen splitter still forces the rendering and synthesis of the damaged signal according to the preset high resolution or high frame rate parameters. This results in inter-frame tearing, artifacts, or stuttering in the output integrated display, reducing the rendering quality of the display and leading to a poor user experience. Summary of the Invention

[0005] This application provides a method, system, medium, and product for integrating multiple signal sources in a reconnaissance vehicle, which is used to improve the rendering quality of the displayed image under dynamic vehicle conditions.

[0006] In a first aspect, this application provides a multi-signal source integrated display method for a reconnaissance vehicle, applied to a display system. The method includes: responding to the access trigger of multiple signal sources, reading video stream data and sensor data of the signal source devices in parallel through a communication interface to obtain a raw heterogeneous data stream; obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset working condition threshold table, and obtaining rendering control instructions; executing the rendering control instructions to decode and resample the raw heterogeneous data stream to generate independent layer objects that conform to the image scaling ratio and frame rate parameters; calling a preset screen layout coordinate file to map multiple independent layer objects to the same canvas coordinate system for pixel overlay to generate a single-frame integrated display image; and writing the single-frame integrated display image into the video memory frame buffer to drive the display terminal to output the single-frame integrated display image.

[0007] In the above embodiments, the display system acquires the vehicle's dynamic operating conditions and determines the corresponding rendering parameters, and performs adaptive rendering processing on the heterogeneous data stream. This avoids screen tearing and stuttering caused by fixed parameter rendering under dynamic operating conditions, and improves the rendering quality of the integrated display screen.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the step of obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset operating condition threshold table, and obtaining rendering control instructions specifically includes: reading the data packet header information of the original heterogeneous data stream and extracting the type identification code of each signal source; obtaining the vehicle's current driving speed and network bandwidth, determining the operating condition level range corresponding to the driving speed and network bandwidth according to the preset operating condition threshold table; and matching the corresponding image scaling ratio and frame rate parameters for signal sources with different type identification codes according to the operating condition level range, thereby obtaining rendering control instructions.

[0009] In the above embodiments, the display system identifies the signal source type and matches differentiated rendering parameters to different types of signals according to the operating condition level, thereby optimizing resource allocation and ensuring the display quality of key information while also taking into account the overall smoothness of system operation.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of executing rendering control instructions to decode and resample the original heterogeneous data stream to generate an independent layer object conforming to the image scaling ratio and frame rate parameters specifically includes: decoding the original heterogeneous data stream to obtain the pixel matrix data of the current frame; calculating the pixel difference between the corresponding pixel values ​​of the pixel matrix data of the current frame and the cached data of the previous frame; when the pixel difference exceeds a preset variation range, performing interpolation smoothing operation on the pixel matrix data during the pixel resampling process to obtain a smoothed pixel matrix; and cropping and recombining the smoothed pixel matrix according to the image scaling ratio to obtain an independent layer object.

[0011] In the above embodiments, the display system calculates the inter-frame pixel difference and performs interpolation smoothing operations on areas with drastic pixel changes, effectively suppressing block effects and artifacts that may be introduced by rapid changes in the image or pixel resampling, and generating a visually smoother layer object.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of reading video stream data and sensor data of the signal source devices in parallel through the communication interface in response to the access trigger of multiple signal sources to obtain the original heterogeneous data stream, the method further includes: extracting the timestamps of each signal in the original heterogeneous data stream; calculating the offset between the timestamps and the system reference clock; adjusting the read / write pointer positions of each signal in the memory queue according to the offset to align the timing of the multiple data streams; and transmitting the timing-aligned heterogeneous data stream to the decoding unit.

[0013] In the above embodiments, the display system achieves time synchronization of multiple heterogeneous data streams by extracting and aligning the timestamps of each signal, ensuring the consistency of video images and associated sensor data in the time dimension, avoiding information mismatch, and ensuring the accuracy of situational awareness.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, after obtaining the current vehicle speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the vehicle speed and network bandwidth according to a preset working condition threshold table, and obtaining the rendering control instruction, the method further includes: calculating the vehicle acceleration based on the current vehicle speed; when it is determined based on the vehicle acceleration that the vehicle deceleration exceeds a preset emergency stop deceleration threshold, changing the rendering parameters in the rendering control instruction to preset highest quality rendering parameters, and continuing for a preset duration after the vehicle speed drops to zero.

[0015] In the above embodiments, the display system monitors the vehicle deceleration and forces a switch to the highest quality rendering mode when an emergency stop occurs, ensuring that operators can obtain the clearest on-site view for decision-making in emergency situations, thereby improving operational safety in special scenarios.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of writing the single-frame integrated display image into the video memory frame buffer and driving the display terminal to output the single-frame integrated display image, the method further includes: monitoring the refresh rate of the video memory frame buffer; when the refresh rate is lower than a preset active threshold, sending a heartbeat detection signal to the corresponding signal source port; when no feedback data of the heartbeat detection signal is received within a preset waiting period, loading preset abnormal warning bitmap data; and using the abnormal warning bitmap data to cover the pixel area of ​​the corresponding layer object in the canvas coordinate system.

[0017] In the above embodiments, the display system actively identifies signal interruption faults by monitoring the frame buffer refresh frequency and combining it with a heartbeat detection mechanism. It can promptly warn operators of signal loss in a visual manner, avoiding the risk of misjudging a still image as a valid real-time signal.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset working condition threshold table, and obtaining the rendering control instruction, the method further includes: real-time monitoring of user interaction focus in multiple display windows to determine the main task window and at least one auxiliary task window; generating an asymmetric rendering instruction when the rendering control instruction indicates that the rendering quality needs to be reduced; the asymmetric rendering instruction includes a first rendering parameter for the main task window and a second rendering parameter for the auxiliary task window; the rendering quality corresponding to the second rendering parameter is lower than the rendering quality corresponding to the first rendering parameter.

[0019] In the above embodiments, the display system distinguishes between primary and secondary task windows by monitoring user interaction focus. When it is necessary to reduce rendering quality, asymmetric rendering is performed, and limited rendering resources are allocated to the primary task window first, thus ensuring the display quality and smoothness of the core interactive area.

[0020] In a second aspect, embodiments of this application provide a display system comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the display system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a display system, cause the display system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a display system, cause the display system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the display system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. Due to the adoption of a dynamic working condition-aware adaptive adjustment mechanism for rendering parameters, rendering is not performed directly after the response signal is received. Instead, the system first acquires the vehicle's speed and network bandwidth, which characterize the vehicle's dynamic working condition. Therefore, the display system can map the real-time acquired speed and bandwidth data into specific image scaling ratios and frame rate parameters based on a preset working condition threshold table, and generate rendering control commands. This enables intelligent and real-time optimization of the rendering quality of multi-signal source integrated displays in dynamically changing environments such as vehicle movement and communication.

[0026] 2. Due to the adoption of a differentiated rendering strategy based on signal source type, after acquiring the original heterogeneous data stream, the system first reads its data packet header information to extract the type identification code of each signal, which is used to distinguish the nature of the signal source (e.g., main reconnaissance video, auxiliary environmental perception video, sensor data panel, etc.). Therefore, after determining the overall operating condition level range based on the vehicle speed and network bandwidth, the display system does not use uniform rendering parameters for all signal sources. Instead, based on the operating condition level range and combined with the type identification code of each signal source, it matches the most suitable image scaling ratio and frame rate parameters for each signal source, prioritizing the display quality of the core reconnaissance images. This allows limited computing and bandwidth resources to be used where they are most critical, improving the overall operational efficiency of the reconnaissance vehicle in complex environments.

[0027] 3. Due to the adoption of an asymmetric rendering mechanism based on user interaction focus, the display system monitors the windows through which the user interacts via input devices such as mouse and touch screen in real time. This dynamically determines the currently active main task window and the remaining auxiliary task windows. Therefore, when the display system determines that the overall rendering quality needs to be reduced based on the vehicle's operating conditions, a higher-quality first set of rendering parameters is applied to the main task window that the user is focusing on, while a lower-quality second set of rendering parameters is applied to other inactive auxiliary task windows. This effectively reduces the overall rendering load of the display system and improves the efficiency of human-computer interaction without sacrificing the user's current core task experience. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a multi-signal source integrated display method for a detection vehicle in an embodiment of this application;

[0029] Figure 2 This is another flowchart illustrating the multi-signal source integrated display method for the detection vehicle in this application embodiment;

[0030] Figure 3 This is a schematic diagram of the physical device structure of a display system in an embodiment of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] In the specific implementation scenario of this application, all the proper nouns and technical terms involved revolve around the multi-signal source integrated display task of the special mobile platform of the reconnaissance vehicle. For example, the reconnaissance vehicle itself is a highly integrated mobile operation platform. Its special characteristic lies in the need to perform reconnaissance and detection tasks while in motion, which means that its display system must cope with dynamic operating conditions such as driving vibration and network fluctuations. Multiple signal sources refer to various information acquisition devices connected to the platform, including but not limited to high-definition video transmitted by drones, vehicle-mounted panoramic cameras, video from reconnaissance robots that can penetrate dangerous areas, and various environmental sensor data such as temperature, humidity, and gas concentration. The raw heterogeneous data streams generated by these signal sources are diverse, and the heterogeneity is reflected in the different data formats (such as H.264 and H.265 video streams, and sensor data in JSON and XML formats), resolutions, and frame rates. The communication interface in the processing flow is the physical or logical port that receives these data streams, such as an Ethernet port or a wireless network card. After reading the data stream, the display system determines the current operating condition based on the driving speed provided by the vehicle's CAN bus and the network bandwidth assessed by the network monitoring module. It then queries a pre-set operating condition threshold table—a pre-defined configuration table that associates speed and bandwidth ranges with rendering parameter levels. The resulting rendering control instructions are the core of the entire adaptive display method, instructing subsequent processing units on the appropriate image scaling and frame rate parameters. Decoding is the process of restoring compressed video stream data to its original pixel data. Pixel resampling is a technique that recalculates and rearranges pixels during image scaling to adapt to new resolutions. Independent layer objects refer to independent graphics units formed in memory after decoding and resampling of each signal source; these units can be operated independently. Finally, the display system calls the screen layout coordinate file (a configuration file defining the position and size of each layer on the screen), overlays these layers pixel-by-pixel in the canvas coordinate system to form a single-frame integrated display image, writes it to the video memory frame buffer, and finally presents it to the display terminal. This series of processing logics ensures that data from different sources and in different formats can be presented to the operator with optimal rendering quality in a unified and stable manner, even in dynamically changing and harsh environments.

[0034] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a multi-signal source integrated display method for a detection vehicle in an embodiment of this application.

[0035] S101, In response to the access trigger of multiple signal sources, the video stream data and sensor data of the signal source devices are read in parallel through the communication interface to obtain the original heterogeneous data stream.

[0036] In this context, the access triggering of multiple signal sources refers to the event detected by the display system when a physical connection (such as inserting a network cable) or logical connection (such as successful pairing of wireless devices) is established. The communication interface refers to the hardware port and protocol stack used for data transmission, such as an Ethernet interface or a Wi-Fi module. Raw heterogeneous data streams refer to unprocessed collections of data from different devices (such as drones and sensors), varying in format and characteristics, including video streams and structured sensor data.

[0037] Specifically, when the operator on the reconnaissance vehicle activates a new reconnaissance device or sensor and connects it to the vehicle network, the main control program of the display system will listen for the corresponding network port or service discovery event and identify it as an access trigger. Once triggered, the display system will allocate an independent I / O processing thread for the new signal source, establish a data connection through the specified communication interface (such as IP address and port), and begin to receive data packets from the signal source and other connected signal sources in parallel and continuously, forming a raw data stream set containing multiple data formats and sources, which is stored in the receive buffer in memory.

[0038] In some embodiments, network jitter can cause data packets to arrive out of order or be lost. To address this, the display system implements a reordering and integrity verification mechanism at the application layer. Upon receiving each data packet, the display system checks its sequence number. Out-of-order packets are first stored in a temporary buffer for sorting; if a non-contiguous sequence number is detected, packet loss is considered, and a policy is used to determine whether to request retransmission (if supported by the protocol) or to discard the frame to ensure real-time performance.

[0039] S102. Obtain the vehicle's current driving speed and network bandwidth, determine the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to the preset working condition threshold table, and obtain the rendering control instructions.

[0040] The vehicle's current speed refers to the real-time speed value obtained through the onboard CAN bus or GPS module. Network bandwidth refers to the effective data transmission rate of the communication link between the display system and each signal source, which can be estimated through periodic detection. The preset operating condition threshold table is a two-dimensional lookup table stored in the configuration file, with its rows and columns representing speed and bandwidth ranges, respectively, and the table entries representing the corresponding rendering parameters. The rendering control instructions are data structures containing specific scaling ratios and frame rate values.

[0041] Specifically, a status monitoring module within the display system periodically (e.g., once per second) queries the vehicle bus interface module and network monitoring module via internal interfaces to obtain the latest driving speed and assessed network bandwidth values. After obtaining these two key parameters, the display system queries a preset operating condition threshold table. For example, the threshold table might define a speed <10km / h and bandwidth >50Mbps as excellent operating conditions, corresponding to a scaling ratio of 1.0 and a frame rate of 30fps; while a speed >60km / h or bandwidth <10Mbps is considered poor operating conditions, corresponding to a scaling ratio of 0.5 and a frame rate of 15fps. The display system matches the current speed and bandwidth values ​​to the corresponding interval, extracts the appropriate image scaling ratio and frame rate parameters, and encapsulates these parameters into a rendering control instruction for use in subsequent steps.

[0042] It's important to note that the pre-set operating condition threshold table is not simply based on empirical settings, but rather on optimization results derived from a series of rigorous experimental tests and data analysis. First, a complete detection vehicle display system is built in a laboratory or actual road test environment, and representative multi-channel signal sources are connected. The testing team will systematically simulate or create different combinations of operating conditions. For example, using an onboard power simulator, the vehicle is precisely controlled to travel at different speeds such as 5km / h, 30km / h, 60km / h, and 90km / h, while a network impairment tester simulates different levels of network bandwidth such as 100Mbps, 50Mbps, 20Mbps, and 5Mbps. Under each "speed-bandwidth" combination, the system attempts to traverse multiple sets of rendering parameters (image scaling ratio, frame rate) within a broad range (e.g., image scaling ratio from 1.0 to 0.3, frame rate from the source frame rate to 5fps). For each set of rendering parameters, the system automatically records key performance indicators (KPIs), including CPU / GPU utilization, end-to-end latency (time from signal source acquisition to screen display), decoding queue length, and records the output screen. Meanwhile, multiple trained testers subjectively scored the recorded footage (MOS score), with scoring criteria covering sharpness, smoothness, and the presence of artifacts or tearing. After testing, KPI data and MOS score data under all conditions were analyzed in a multi-dimensional correlation to find a balance: that is, under specific conditions, a combination of rendering parameters that minimizes system resource consumption and latency while ensuring a subjective visual experience (MOS score not falling below a certain acceptable threshold, such as 3.5). This parameter combination becomes the "preset" item for that condition in the threshold table. The final threshold table is essentially a set of quantitative strategies that achieve the best trade-off between satisfying user experience and ensuring system stability.

[0043] In some embodiments, there may be instances where sudden and drastic fluctuations in speed or bandwidth data lead to frequent switching of rendering parameters. To address this, the display system introduces a data smoothing and hysteresis processing mechanism. Instead of directly using the instantaneously acquired speed and bandwidth values, the display system performs a moving average calculation on their values ​​over a certain time window (e.g., the past 5 seconds) and uses the smoothed value for table lookup. Simultaneously, a hysteresis threshold is set when switching operating conditions; that is, rendering parameter changes are only executed after the smoothed operating condition value significantly crosses the boundary and persists for a period of time, thus avoiding unnecessary image quality jitter caused by brief disturbances.

[0044] S103. Execute rendering control instructions to decode and resample the original heterogeneous data stream, generating independent layer objects that conform to the image scaling ratio and frame rate parameters.

[0045] Executing rendering control instructions refers to the decoding and resampling units adjusting their processing behavior according to the parameters in the instructions. Decoding is the process of restoring compressed video data (such as H.265 bitstreams) into frames of YUV or RGB pixel data. Pixel resampling processing enlarges or reduces the decoded pixel array according to the image scaling ratio in the instructions to generate an image of the target size. An independent layer object is the processed memory image data representing a single signal source and conforming to the target parameters.

[0046] Specifically, the raw heterogeneous data stream retrieved from the memory queue, if it is a video stream, is sent to the decoder. When decoding each frame, the decoder refers to the frame rate parameter in the rendering control instructions. For example, if the instructions require 15fps, but the raw bitstream is 30fps, the decoder will adopt an inter-frame decoding strategy, discarding the next frame after decoding the previous one to reduce the frame rate. Each decoded frame of pixel data is then sent to the pixel resampling module. This module reads the scaling ratio in the instructions and uses an appropriate algorithm (such as bilinear interpolation) to scale the image, generating a new pixel array. This pixel array, meeting the frame rate and size requirements of the instructions, constitutes an independent layer object, awaiting subsequent compositing.

[0047] In some embodiments, this step can be implemented in several ways: Optionally, a hardware-accelerated decoder (such as NVDEC) can be used to offload the decoding task to the GPU, reducing the CPU load. The decoded data is directly stored in the GPU's video memory, avoiding data copying between the CPU and GPU. Optionally, during the pixel resampling stage, the interpolation algorithm is dynamically selected based on the image quality requirements. For example, Lanczos interpolation, which has a higher computational cost but better performance, can be used in high-quality rendering mode, while bilinear interpolation, which is faster, can be used in low-quality mode. It is understood that other methods can also be used to implement this processing, and no limitation is made here.

[0048] In some embodiments, insufficient decoding capabilities may lead to accumulated screen latency. To address this, the display system monitors the length of the decoding queue. If too many data frames accumulate in the queue, exceeding a preset threshold, it indicates that the processing speed cannot keep up with the input speed. In this case, the display system will proactively discard some older video frames in the queue, decoding only the most recent frames. This strategy is called frame chasing, and its purpose is to sacrifice some screen continuity to ensure the real-time performance of the displayed image and prevent the operator from seeing severely outdated images.

[0049] S104. Call the preset screen layout coordinate file, map multiple independent layer objects to the same canvas coordinate system for pixel overlay, and generate a single-frame integrated display image.

[0050] The screen layout coordinate file is a configuration file that defines the layout of multiple screens, recording the starting coordinates and size of each child window on the screen. The canvas coordinate system refers to a virtual two-dimensional plane used for the final composite image. Pixel overlay refers to the process of drawing individual layer objects onto the canvas according to the positions and sizes specified in the layout file. The single-frame integrated display image is a complete image containing all child screens, generated after all layers have been overlaid.

[0051] Specifically, after each round of preparation (i.e., all signal sources have generated a new frame layer object), the compositing module of the display system first loads and parses the preset screen layout coordinate file. This file may define a four-screen layout. Next, the compositing module creates a blank canvas with the same resolution as the display terminal. Then, it iterates through all the individual layer objects, looks up their corresponding position information in the layout file based on their signal source ID, and then calls functions from a graphics library (such as OpenGL) to draw the pixel data of the layer object onto a rectangular area at specified coordinates on the canvas. After all layer objects have been drawn, a complete single-frame integrated display image containing multiple sub-pictures is formed on the canvas.

[0052] In some embodiments, there may be situations where users need to temporarily adjust the layout. To address this, the display system supports dynamic layout switching. The display system pre-loads various layout files (such as four-screen split, picture-in-picture, etc.). When a user's switching command is received (such as double-clicking a child window), the compositing module immediately loads the layout file corresponding to the command and uses the new layout coordinates to map and overlay layer objects during the next frame compositing, thereby achieving seamless layout switching.

[0053] S105. Write the single-frame integrated display image into the video memory frame buffer and drive the display terminal to output the single-frame integrated display image.

[0054] The video memory frame buffer is a dedicated area of ​​memory on the graphics card used to store image data that will be displayed on the screen. Display terminal output refers to the process of displaying the image data from the frame buffer onto the physical screen through the operating system and the graphics card driver.

[0055] Specifically, once the single-frame integrated display image generated in step S104 is ready, the final step of the graphics rendering pipeline performs a buffer swapping operation. Under the double buffering mechanism, rendering operations always occur in a back buffer, while the screen currently displays the contents of the front buffer. The buffer swapping operation involves exchanging the pointers of the recently rendered back buffer and the front buffer. Thus, when the next vertical sync signal arrives, the graphics card begins reading the contents of the new front buffer and scans it line by line to output to the monitor, allowing the user to see the updated integrated image.

[0056] In some embodiments, the output of this step can be achieved in several ways: Optionally, vertical synchronization can be enabled, meaning that the buffer swapping operation must wait until the display completes a frame refresh before it can proceed; this method prevents screen tearing. Optionally, vertical synchronization can be disabled, and the buffers are swapped immediately after rendering; this method reduces latency, but may cause screen tearing if the rendering frame rate does not match the display refresh rate. It is understood that other methods can also be used to achieve the output, and these are not limited here.

[0057] In some embodiments, the rendering speed may exceed the display refresh rate, resulting in unnecessary waste of computing resources. To address this, the display system implements a frame rate controller. This controller limits the maximum execution frequency of the rendering loop to not exceed the physical refresh rate of the display terminal (e.g., 60Hz). Specifically, after rendering and submitting each frame, the time spent processing that frame is calculated. If the time is less than the frame cycle, the rendering thread is put to sleep for the corresponding period to conserve computing resources.

[0058] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the multi-signal source integrated display method for the detection vehicle in this application embodiment.

[0059] S201, In response to the access trigger of multiple signal sources, the video stream data and sensor data of the signal source devices are read in parallel through the communication interface to obtain the original heterogeneous data stream.

[0060] Refer to step S101, which will not be repeated here.

[0061] S202. Read the header information of the original heterogeneous data stream and extract the type identification code of each signal source.

[0062] The packet header is a section of data located at the beginning of each network packet, describing the packet content and attributes. The type identifier is a pre-agreed unique code used to distinguish different types of signal sources; for example, 0x01 represents video from a drone's main camera, and 0x02 represents video from a vehicle-mounted wide-angle camera.

[0063] Specifically, after parallel reading of the data stream in step S201, the data is sent to a preprocessing module. This module follows a predefined application layer protocol that specifies the structure of the data packets, where the header always contains a type identifier field. For each received data packet, the preprocessing module first parses its header, locates the type identifier field, and reads its value, i.e., extracts the type identifier code. This identifier code, along with the data packet content, is appended to the data packet as a metadata tag for use in subsequent differentiation processing steps.

[0064] In some embodiments, this extraction step can be implemented in several ways: Optionally, a fixed offset method can be used, where the protocol specifies that the type identifier code is always in a fixed position in the packet header, and the parser can directly read this position; alternatively, a TLV (Type-Length-Value) format packet header can be used, where the packet header consists of a series of fields for tags, length, and value. During parsing, the packet header needs to be traversed to find the tag of type signal source, and then its value is read. This method has good scalability. It is understood that other methods can also be used to implement extraction, which are not limited here.

[0065] In some embodiments, devices from different manufacturers may use different proprietary protocols, resulting in inconsistent type identification code formats. To address this, the display system designs an adapter layer. When a new device is connected, a corresponding protocol adapter plugin needs to be configured for it. This plugin is responsible for converting the device's proprietary protocol header into the unified standard format used within the display system. When data streams arrive, the display system first loads the appropriate adapter based on the signal source, and the adapter performs header parsing and type identification code extraction.

[0066] S203. Obtain the vehicle's current driving speed and network bandwidth, and determine the corresponding operating condition level range based on the preset operating condition threshold table.

[0067] The operating condition level range is a comprehensive rating of the vehicle's operating environment and communication status, for example, divided into four levels: excellent, normal, restricted, and severe. The output of this step is not a direct rendering parameter, but a macroscopic level.

[0068] Specifically, the system's status monitoring module also periodically acquires driving speed and network bandwidth. The operating condition threshold table here doesn't define specific parameters, but rather operating condition levels. For example, the table might define: speed < 10 km / h and bandwidth > 50 Mbps corresponds to an excellent level; speed >= 60 km / h or bandwidth < 10 Mbps corresponds to a severe level. The display system matches the current speed and bandwidth values ​​against this table to determine a unique operating condition level range, such as normal. This level range will serve as the basis for the next step of differentiated parameter matching.

[0069] In some embodiments, this step can be determined in several ways: Optionally, a simple two-dimensional matrix lookup table method can be used, with the speed and bandwidth ranges serving as row and column indices of the matrix to directly look up the corresponding level; alternatively, a scoring mechanism can be used, assigning different scores to different speed and bandwidth ranges, then adding the scores together to obtain a total score, and determining the final operating condition level based on the range in which the total score falls. It is understood that other methods can also be used to determine the level, and this is not limited here.

[0070] In some embodiments, there may be a problem where the operating condition level frequently jumps between two adjacent levels. To address this, the display system introduces a hysteresis mechanism when determining the operating condition level. For example, upgrading from normal to excellent requires the operating condition parameters to continuously meet the excellent condition conditions for more than 3 seconds, while downgrading from excellent to normal may be executed immediately as soon as the parameters fall into the normal range. This asymmetric switching threshold design can effectively suppress level jitter caused by data boundary fluctuations, making the determination of the operating condition level more stable.

[0071] S204. Based on the operating condition level range, match the corresponding image scaling ratio and frame rate parameters for signal sources with different types of identification codes to obtain rendering control instructions.

[0072] This step is the core of differentiated rendering. It receives the operating condition level range and the type identification code of each signal source determined in the previous step, and then queries a more complex parameter configuration table to generate customized rendering parameters for each signal source.

[0073] Specifically, the display system maintains a detailed differentiated rendering strategy table. This table is a three-dimensional mapping table, with inputs of (condition level, signal source type) and outputs of (image scaling ratio, frame rate parameters). For example, when the condition level determined in step S203 is severe, the display system iterates through all signal sources. For signal sources with the type identifier code of primary reconnaissance video, the strategy table is consulted to obtain parameters that should be (scaling ratio 0.7, frame rate 20fps); while for signal sources with the type identifier code of auxiliary environment video, the parameters are obtained as (scaling ratio 0.4, frame rate 10fps). The display system combines the parameters matched for each signal source to form a rendering control instruction containing multiple sets of parameters.

[0074] In some embodiments, the matching process can be implemented in several ways: Optionally, the strategy table can be a multi-level nested JSON file, and the program can find the corresponding parameters level by level through key-value lookup; alternatively, the strategy can be defined as a set of scripts (such as Lua scripts), which take the operating condition level and signal type as input, and return the parameters after execution. This approach provides greater flexibility. It is understood that other methods can also be used to achieve matching, and no limitation is made here.

[0075] In some embodiments, certain signal sources (such as critical alarm signals) must maintain the highest display quality regardless of operating conditions. To address this, the differentiated rendering strategy table supports priority or exemption flags. For signal source types marked as highest priority, the rendering parameters for all corresponding operating condition levels in the strategy table are set to the highest quality. This ensures that during parameter matching, these signal sources always receive optimal rendering resources, no matter how severe the current operating conditions.

[0076] In some embodiments, after generating the rendering control command in step S204, the display system calculates the driving acceleration based on the vehicle's current driving speed in order to cope with sudden emergency braking situations. When it is determined that the vehicle's deceleration exceeds a preset emergency braking deceleration threshold based on the driving acceleration, the rendering parameters in the rendering control command are changed to preset highest quality rendering parameters, and this continues for a preset duration after the driving speed drops to zero.

[0077] Among them, driving acceleration is the rate of change of driving speed with respect to time, which can be obtained by differential calculation of the speed value. Emergency braking deceleration threshold is a preset negative acceleration value used to determine whether an emergency braking event has occurred. Highest quality rendering parameters refer to parameter combinations without any degradation processing (e.g., scaling ratio 1.0, frame rate consistent with the source).

[0078] Specifically, the display system's status monitoring module, in addition to acquiring the current speed, also caches historical speed data. By comparing the current speed with the speed at the previous sampling moment, the vehicle's acceleration can be calculated. When the calculated acceleration is a large negative value, and its absolute value exceeds the preset emergency braking deceleration threshold (e.g., -5 m / s²), the display system determines that the vehicle is undergoing emergency braking. At this time, the display system immediately generates a highest priority instruction, forcibly overriding the current rendering control instruction determined by the operating condition. This instruction requires all signal sources to render using the preset highest quality rendering parameters. This high-quality rendering mode will be maintained until the vehicle speed drops to zero, and will continue for a preset duration (e.g., 10 seconds) after the speed reaches zero to ensure that the operator has sufficient time to observe the surrounding environment after the emergency stop.

[0079] In some embodiments, this step can be triggered in several ways: Optionally, accelerometer data transmitted via the CAN bus can be used directly, which is more accurate and timely than speed differential calculation; alternatively, the status of the brake pedal sensor can be considered, i.e., the highest quality rendering mode is triggered only when both a deceleration exceeding the threshold and a brake pedal depressor signal are detected, to avoid misjudgments caused by steep downhill conditions. It is understood that other methods can also be used to trigger this process, and this is not limited here.

[0080] In some embodiments, a sudden switch from low-quality rendering mode to the highest quality mode may cause a brief system stutter due to the surge in decoding and rendering load. To address this, the display system reserves some computing resources for the highest quality rendering mode. During normal low-power or low-quality rendering modes, some GPU cores or decoder hardware are idle or in a low-power state. When an emergency stop event is triggered, the display system immediately wakes up these reserve resources, enabling them to fully engage in high-quality rendering tasks. This smoothly handles the sudden increase in load, ensuring a seamless switching process and guaranteeing timely rendering of images at critical moments.

[0081] S205. Monitor user interaction focus in multiple display windows in real time to determine the main task window and at least one secondary task window.

[0082] In this context, the user interaction focus refers to the interface element that the user is currently interacting with through an input device (such as a mouse or touchscreen), typically represented by an active window. The main task window is the window that currently has focus, while the secondary task windows refer to all other windows that do not have focus.

[0083] Specifically, a UI event listening module in the display system continuously captures window events from the operating system, such as mouse clicks and window activation / deactivation. When a window is activated by the user (e.g., by clicking its title bar), the module identifies that window as the primary task window and all other visible windows as secondary task windows. This identification is dynamic and updates in real time as the user's focus shifts. For example, if the user switches from window A to window B, window A changes from a primary task window to a secondary task window, while window B changes from a secondary task window to a primary task window.

[0084] In some embodiments, this step can be monitored in several ways: Optionally, the handle of the currently active window can be periodically obtained by polling the API provided by the operating system and compared with the window list managed by the display system; alternatively, an event callback mechanism can be used to register a hook function for window events with the operating system. When the focus changes, the operating system will actively call this function to notify the display system to update the status, which provides a more timely response. It is understood that other methods can also be used to implement focus monitoring, and this is not limited here.

[0085] In some embodiments, a user may focus on multiple windows simultaneously, but only one window can receive focus. To address this, the display system can introduce the concept of focus. Besides designating the focused window as the primary task window, the display system also calculates a focus score for other windows based on the user's recent operation history (such as mouse hover time and window dragging frequency). Secondary task windows with higher scores can achieve slightly higher rendering quality in subsequent asymmetric rendering, enabling more refined resource allocation.

[0086] S206. When the rendering control instruction indicates that the rendering quality needs to be reduced, generate an asymmetric rendering instruction.

[0087] The rendering control instruction indicating a need to reduce rendering quality means that the instruction generated in step S204 contains rendering parameters lower than the standard quality. Asymmetric rendering instructions are a special data structure that defines different sets of rendering parameters for the main task window and the auxiliary task window, respectively.

[0088] Specifically, this step is a reprocessing of the initial rendering control instructions generated in step S204. When executing this step, the display system first checks the initial rendering control instructions. If all parameters in the instructions are high-quality, the instructions are directly passed through. If the instructions indicate a need to reduce rendering quality (e.g., a severe condition level), asymmetric rendering logic is triggered. Based on the primary and secondary task windows determined in step S205, the display system splits the initial instructions and reconstructs them into a single asymmetric rendering instruction. In this new instruction, the signal source associated with the primary task window is assigned first rendering parameters (usually the parameters from the initial instructions or slightly optimized parameters), and lower-quality second rendering parameters are assigned to the signal sources associated with all secondary task windows (e.g., further reducing the frame rate and scaling ratio based on the initial instruction parameters).

[0089] In some embodiments, this step can be implemented in several ways: Optionally, a fixed degradation strategy can be maintained, for example, the frame rate and scaling ratio of the second rendering parameter can be fixed at 50% of the first rendering parameter; alternatively, the degradation magnitude can be dynamically calculated based on the number of auxiliary task windows, with more auxiliary task windows resulting in lower quality of the second rendering parameter, to ensure the stability of the total system load. It is understood that other methods can also be used to generate asymmetric instructions, which are not limited here.

[0090] In some embodiments, frequent switching of the main task window can cause drastic fluctuations in image quality. To address this, the display system introduces a smooth transition mechanism. When the focus switches from window A to window B, the display system does not immediately reduce the quality of window A to the lowest level and increase the quality of window B to the highest level. Instead, it smoothly and linearly adjusts the rendering parameters of the two windows within a short time window (e.g., 0.5 seconds), making the quality change visually imperceptible and improving the continuity of the user experience.

[0091] S207. Execute rendering control instructions to decode and resample the original heterogeneous data stream, generating independent layer objects that conform to the image scaling ratio and frame rate parameters.

[0092] This step is similar in function to step S103, but it is based on more complex asymmetric rendering instructions that may be reconstructed through step S206.

[0093] Specifically, when processing the data stream of each signal source, the decoding and resampling units no longer use uniform rendering parameters. Instead, they look up specific rendering parameters corresponding to the signal source ID or its associated window (main / secondary task window) from the rendering control instructions. For example, for signal source A associated with the main task window, the decoder and resampler will process it using the first rendering parameters specified in the instructions (such as a scaling ratio of 0.8 and a frame rate of 25fps); while for signal source B associated with the secondary task window, they will process it using a second rendering parameter with lower quality (such as a scaling ratio of 0.5 and a frame rate of 10fps). In this way, the same processing flow generates independent layer objects of varying quality for different signal sources based on the differences in the instructions.

[0094] In some embodiments, the image quality of the secondary task window may be so low that background information becomes completely unrecognizable. To address this, the display system sets a quality minimum for the second rendering parameters. That is, regardless of the operating conditions or the number of secondary task windows, the image scaling ratio and frame rate must not fall below a preset minimum threshold (e.g., scaling ratio 0.2, frame rate 5fps). This measure ensures that even non-focus windows provide basic, recognizable image information, preventing complete information loss.

[0095] In some embodiments, when performing pixel resampling processing in step S207, in order to further improve the visual quality of dynamic images, the display system decodes the original heterogeneous data stream to obtain the pixel matrix data of the current frame; calculates the pixel difference between the corresponding pixel values ​​of the current frame's pixel matrix data and the cached data of the previous frame; when the pixel difference exceeds a preset variation range, performs interpolation smoothing operation on the pixel matrix data during pixel resampling processing to obtain a smoothed pixel matrix; and crops and reassembles the smoothed pixel matrix according to the image scaling ratio to obtain an independent layer object.

[0096] Pixel array data refers to the two-dimensional array of pixel values ​​obtained after decoding. Pixel difference is the numerical difference between two pixels in the color space. Interpolation smoothing is an image processing algorithm that smooths image transitions by inserting calculated new pixel values ​​between pixels, such as bicubic interpolation. Smoothed pixel array data is the pixel data after this operation.

[0097] Specifically, after the decoder outputs a frame of pixel data, the display system compares it with the previous frame's data stored in the buffer. This comparison is performed pixel-by-pixel, calculating the sum of the absolute values ​​of the differences between the R, G, B or Y, U, V components of the corresponding pixel. If this difference exceeds a preset threshold for variation (indicating a significant change in the pixel, possibly at the edge of a dynamic object), the display system considers that area to require smoothing. In subsequent pixel resampling (scaling), for areas requiring smoothing, the display system uses a higher-order interpolation algorithm (such as B-spline interpolation) to calculate the new pixel value, while for static areas with minimal change, it may use a less computationally intensive algorithm (such as nearest neighbor interpolation). After processing, the image is cropped and reassembled according to the scaling ratio required by the rendering control instructions to generate the final independent layer object.

[0098] It's important to note that the core of performing interpolation smoothing lies in implementing content-adaptive image scaling quality control. Specifically, when the decoder outputs a frame of pixel data in YUV or RGB format, the system doesn't simply calculate the brightness difference of individual pixels. To more accurately assess the motion and changes of macroblocks (e.g., a 16x16 pixel region), the system calculates the "Sum of Absolute Differences" (SAD) between each macroblock in the current frame and the corresponding macroblock in the previous frame. The SAD value is calculated by summing the absolute values ​​of the differences in the brightness components (Y components) of all corresponding pixels within the two macroblocks. This SAD value effectively quantifies the degree of inter-frame change in that region. The system presets a "variation amplitude" SAD threshold, which is also determined experimentally: by observing a large number of video samples, a critical SAD value that can effectively distinguish between static / slow-moving regions and high-speed motion / scene transition regions is found. When the SAD value of a macroblock exceeds this threshold, the system determines that region as a "dynamic region." In the subsequent pixel resampling (scaling) stage, the system employs a hybrid interpolation strategy: for macroblocks marked as "dynamic regions," it calls interpolation algorithms with higher computational complexity but better anti-aliasing and blurring effects, such as bicubic or Lanczos interpolation, to ensure the clarity and edge smoothness of moving images; while for "static regions" with SAD values ​​below a threshold, it uses faster bilinear or even nearest-neighbor interpolation algorithms, thus significantly saving valuable computational resources without sacrificing the visual quality of static backgrounds. This SAD-based dynamic region recognition and differential interpolation technology enables intelligent allocation of rendering costs for different regions within a single frame.

[0099] In some embodiments, after receiving the original heterogeneous data stream in step S201 and before sending it to the decoding unit in step S207, the display system extracts the timestamps of each signal in the original heterogeneous data stream to ensure the timing consistency of data from different sources; calculates the offset between the timestamps and the system reference clock; adjusts the read / write pointer positions of each signal in the memory queue according to the offset to align the timing of the multiple data streams; and transmits the timing-aligned heterogeneous data stream to the decoding unit.

[0100] The timestamp is the time information appended to the data packet by the signal source device when acquiring or transmitting data. The system reference clock is a unified, high-precision time reference within the system. The offset is the difference between the signal timestamp and the system reference clock at the same moment. The read / write pointer of the memory queue is a variable used to indicate the location of data read / write operations.

[0101] Specifically, when heterogeneous data streams arrive at the display system, they are not immediately sent to the decoding queue. Instead, they first enter a dedicated timing alignment buffer. The display system parses each data packet and extracts its original timestamp (e.g., presentation timestamp, PTS). Simultaneously, the display system records the system reference clock time when the data packet was received. By comparing these two times, the overall offset of the signal stream relative to the system clock can be calculated. This offset is mainly caused by network latency and device clock differences. After calculating the offset for each signal source, the display system selects a reference (e.g., the stream with the largest delay) and then adjusts the read pointers of other signal streams in their respective buffers to align their delays with the reference. For example, if signal A is 100ms ahead of the reference, its read pointer will be moved backward, causing it to wait an additional 100ms before being read. Only after alignment are the data streams sent to the subsequent decoding units.

[0102] It's important to note that achieving precise timing alignment of multiple heterogeneous data streams essentially involves implementing a lightweight network time synchronization and playback buffer management mechanism within the display system. Specifically, when a data packet arrives for each stream (e.g., a drone video stream and a robot sensor data stream), the system records its local system reference clock time (T_arrival) and parses the presentation timestamp (PTS) generated by the source device from the packet. The system maintains a dynamic "clock offset" estimate (Offset_est) for each stream. This estimate is not calculated all at once but is continuously updated using a smoothing filtering algorithm, similar to the clock synchronization concept in Network Time Protocol (NTP). For example, the system can use a sliding window to calculate the average of the PTS - T_arrival for the most recent N data packets to filter out noise caused by network latency jitter in individual data packets, resulting in a relatively stable latency estimate. After the latency estimates for all streams have stabilized, the system selects the stream with the largest latency as the "synchronization reference delay" (Max_Delay). For any non-baseline stream, the additional waiting time required is the latency estimate of Max_Delay itself. This waiting is not achieved through thread sleeping, but rather reflected in the playback logic: the system sets a playback buffer (Jitter Buffer) for each stream and sets a safety bit for the buffer based on the required waiting time. The decoder does not always read data as soon as there is data in the buffer, but waits until the amount of data in the buffer reaches its corresponding safety bit before starting to consume it at a uniform rate. By setting different buffer levels for streams with different latency, the timestamps of the images and data from all streams are logically perfectly aligned at the moment they are sent to the decoding and rendering pipeline, thus ensuring the synchronization of video and data on the final integrated screen.

[0103] In some embodiments, certain sensor data streams may lack timestamps or have inaccurate timestamps. To address this, the display system uses arrival time as an alternative. For data streams without valid timestamps, the display system assigns a local timestamp to the data packet at the moment it arrives at the system's receiving port, using the system's reference clock. Although this timestamp includes network latency, it can be approximated as fixed under relatively stable network conditions. In this way, all data streams (regardless of whether they originally had timestamps) have a unified time stamp, enabling subsequent alignment operations and ensuring system compatibility.

[0104] S208. Call the preset screen layout coordinate file, map multiple independent layer objects to the same canvas coordinate system for pixel overlay, and generate a single-frame integrated display image.

[0105] Refer to step S104, which will not be repeated here.

[0106] S209. Write the single-frame integrated display image into the video memory frame buffer and drive the display terminal to output the single-frame integrated display image.

[0107] Refer to step S105, which will not be repeated here.

[0108] In some embodiments, after the display system outputs the integrated image to the display terminal in step S209, in order to monitor the actual connection status of each signal source, the display system monitors the refresh rate of the video memory frame buffer; when the refresh rate is lower than a preset active threshold, a heartbeat detection signal is sent to the corresponding signal source port; when no feedback data of the heartbeat detection signal is received within a preset waiting period, preset abnormal warning bitmap data is loaded; and the abnormal warning bitmap data is used to cover the pixel area of ​​the corresponding layer object in the canvas coordinate system.

[0109] The refresh rate of the video memory frame buffer refers to the frequency at which the layer area corresponding to a certain signal source is updated by new data. A heartbeat detection signal is a short network data packet used to detect whether the other end is alive. Anomaly warning bitmap data is a pre-stored image containing warning icons (such as red exclamation marks).

[0110] Specifically, a monitoring thread in the display system maintains a refresh counter for each displayed layer object. If a layer is not updated by a new decoded frame for a period of time (e.g., 2 seconds) (i.e., the counter remains unchanged), the display system assumes that the signal source may have been interrupted. At this time, the display system will actively send a heartbeat detection data packet to the IP address and port of the signal source. If no response packet is received from the signal source within a preset waiting period (e.g., 500 milliseconds), the display system determines that the signal source has been definitively lost. Subsequently, the display system will load a preset abnormality warning bitmap, and in the next round of image composition, it will no longer draw the (outdated) image of the signal source, but will instead use the warning bitmap to cover its original pixel area in the canvas coordinate system.

[0111] In some embodiments, the monitoring and alerting of this step can be implemented in several ways: Optionally, in addition to sending heartbeat packets, the status of the underlying Socket connection can be checked, and if the connection has been broken, it can be directly determined as lost; Optionally, the abnormal alert bitmap can contain richer information, such as the signal source name, the time of loss, etc., and this information can be dynamically rendered onto the alert bitmap by the display system. It is understood that other methods can also be used to implement this monitoring and alerting, which are not limited here.

[0112] In some embodiments, there may be situations where the signal is only temporarily interrupted (e.g., a drone briefly enters a signal blind spot and then flies out again), but the display system has already marked it as lost and displayed a warning icon. In response, after determining that the signal is lost, the display system does not close the underlying communication port, but instead enters a background reconnection mode. In this mode, the display system periodically (e.g., every 5 seconds) attempts to resend heartbeat packets or connection requests to the signal source. Once feedback is received or the reconnection is successful, the display system immediately removes the warning bitmap and resumes the normal reception, decoding, and display process of the signal source data, achieving automatic re-display after signal recovery and reducing manual intervention by the operator.

[0113] In this embodiment, by adopting an adaptive rendering parameter adjustment method based on vehicle dynamic conditions and user interaction focus, the display system can dynamically and differentiatedly match the optimal image scaling ratio and frame rate parameters for multiple heterogeneous signal sources according to real-time changes in driving speed, network bandwidth, and user attention window. This effectively solves the problem that existing technologies use fixed hardware logic to force the rendering of high-quality signals under dynamic conditions, resulting in screen tearing and stuttering, or indiscriminately reducing the quality of all screens and affecting the core task experience. As a result, it achieves stable, clear, and smooth integrated display screen under all-weather and all-road conditions for the reconnaissance vehicle, and can intelligently allocate rendering resources centered on the user, significantly improving the system's environmental adaptability, reliability, and human-computer interaction efficiency.

[0114] The display system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the physical device structure of the display system in the embodiments of this application.

[0115] It should be noted that, Figure 3 The structure of the display system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0116] like Figure 3 As shown, the display system includes a CPU 301, which can perform various appropriate actions and processes according to a program stored in ROM 302 or a program loaded into RAM 303 from storage section 308, such as executing the methods described in the above embodiments. RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0117] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including hard disks, etc.; and communication section 309 including network interface cards such as LAN (Local Area Network) cards, modems, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0118] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by CPU 301, it performs the various functions defined in the present invention.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0120] Specifically, the display system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the multi-signal source integrated display method for the detection vehicle provided in the above embodiment.

[0121] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the display system described in the above embodiments; or it may exist independently and not assembled into the display system. The storage medium carries one or more computer programs, which, when executed by a processor of the display system, cause the display system to implement the multi-signal source integrated display method for the detection vehicle provided in the above embodiments.

[0122] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0123] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

Claims

1. A multi-signal source integrated display method for a reconnaissance vehicle, characterized in that, Applied to a display system, the method includes: In response to the access trigger of multiple signal sources, the video stream data and sensor data of the signal source devices are read in parallel through the communication interface to obtain the original heterogeneous data stream; The vehicle's current driving speed and network bandwidth are obtained, and the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth are determined according to the preset working condition threshold table to obtain rendering control instructions. The rendering control instructions are executed to decode and resample the original heterogeneous data stream, generating an independent layer object that conforms to the image scaling ratio and the frame rate parameter; By calling a preset screen layout coordinate file, multiple independent layer objects are mapped to the same canvas coordinate system for pixel overlay to generate a single-frame integrated display image; The single-frame integrated display image is written into the video memory frame buffer, and the display terminal is driven to output the single-frame integrated display image.

2. The method according to claim 1, characterized in that, The steps of obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset working condition threshold table, and obtaining rendering control instructions specifically include: Read the header information of the original heterogeneous data stream and extract the type identification code of each signal source; Obtain the vehicle's current driving speed and network bandwidth, and determine the operating condition level range corresponding to the driving speed and network bandwidth according to a preset operating condition threshold table; Based on the operating condition level range, corresponding image scaling ratios and frame rate parameters are matched for signal sources with different type identification codes to obtain rendering control instructions.

3. The method according to claim 1, characterized in that, The step of executing the rendering control instructions, decoding and resampling the original heterogeneous data stream, and generating an independent layer object that conforms to the image scaling ratio and the frame rate parameter specifically includes: The original heterogeneous data stream is decoded to obtain the pixel matrix data of the current frame; Calculate the pixel difference between the corresponding pixel values ​​in the current frame's pixel dot matrix data and the cached data of the previous frame; When the pixel difference exceeds a preset variation range, an interpolation smoothing operation is performed on the pixel dot matrix data during the pixel resampling process to obtain a smooth pixel dot matrix. The independent layer object is obtained by cropping and recombining the smooth pixel array according to the image scaling ratio.

4. The method according to claim 1, characterized in that, After the step of responding to the access trigger of multiple signal sources and reading the video stream data and sensor data of the signal source devices in parallel through the communication interface to obtain the original heterogeneous data stream, the method further includes: Extract the timestamps of each signal from the original heterogeneous data stream; Calculate the offset between the timestamp and the system reference clock; The read / write pointer positions of each signal in the memory queue are adjusted according to the offset to align the timing of the multiple data streams. The time-aligned heterogeneous data stream is transmitted to the decoding unit.

5. The method according to any one of claims 1-4, characterized in that, After the steps of obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset working condition threshold table, and obtaining rendering control instructions, the method further includes: Calculate the vehicle's acceleration based on its current speed; When the deceleration of the vehicle exceeds a preset emergency stop deceleration threshold based on the driving acceleration, the rendering parameters in the rendering control command are changed to preset highest quality rendering parameters, and this continues for a preset duration after the driving speed drops to zero.

6. The method according to claim 1, characterized in that, After the step of writing the single-frame integrated display image into the video memory frame buffer and driving the display terminal to output the single-frame integrated display image, the method further includes: Monitor the refresh rate of the video memory frame buffer; When the refresh frequency is lower than the preset active threshold, a heartbeat detection signal is sent to the corresponding signal source port; If no feedback data of the heartbeat detection signal is received within the preset waiting period, the preset abnormal warning bitmap data is loaded. The abnormal warning bitmap data is used to cover the pixel area of ​​the corresponding layer object in the canvas coordinate system.

7. The method according to claim 1, characterized in that, After the steps of obtaining the vehicle's current driving speed and network bandwidth, determining the image scaling ratio and frame rate parameters corresponding to the driving speed and network bandwidth according to a preset working condition threshold table, and obtaining rendering control instructions, the method further includes: Real-time monitoring of user interaction focus in multiple display windows to determine the main task window and at least one secondary task window; When the rendering control instruction indicates that the rendering quality needs to be reduced, an asymmetric rendering instruction is generated; the asymmetric rendering instruction includes a first rendering parameter for the main task window and a second rendering parameter for the secondary task window; the rendering quality corresponding to the second rendering parameter is lower than the rendering quality corresponding to the first rendering parameter.

8. A display system, characterized in that, The display system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the display system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the display system, the display system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on a display system, the display system performs the method as described in any one of claims 1-7.

Images