A method, device, equipment and medium for correcting vehicle-mounted screen projection

Abstract

This invention relates to the field of in-vehicle multi-screen interaction technology, and discloses an in-vehicle projection correction method, device, equipment, and medium, comprising: acquiring projection images of a first screen and a second screen at the same time; inputting the projection images of the first screen and the second screen into a preset classification model respectively, and judging the first state of the first screen and the second state of the second screen respectively; comparing the first state and the second state, if they are the same, the projection is judged to be normal and the step of acquiring projection images of the first screen and the second screen at the same time continues; if they are different, the projection is judged to be abnormal, and the projection image of the second screen is corrected based on the projection image of the first screen. This invention identifies the state category of each screen image through a classification model, and automatically completes the correction based on the main screen when the judged state is inconsistent, ensuring the synchronization of multi-screen projection state and improving driving safety.

Description

Technical Field

[0001] This invention relates to the field of in-vehicle multi-screen interaction technology, specifically to a method, device, equipment, and medium for correcting in-vehicle screen projection. Background Technology

[0002] With the development of smart cockpit technology, multi-screen collaboration in vehicles has become a mainstream configuration. The central control screen typically projects navigation or video feeds onto the instrument panel and / or HUD to provide the driver with navigation information. However, in actual use, due to factors such as abnormal data transmission and rendering issues at the projection end, inconsistencies often arise between the projected image displayed on the projection end and the image displayed on the central control screen. Currently, related technologies lack an effective mechanism for proactively correcting the projection end after detecting anomalies. Summary of the Invention

[0003] This invention provides a method, device, equipment, and medium for correcting in-vehicle screen projection, in order to solve the problem of inconsistent display states of various screens affecting driving safety in in-vehicle multi-screen projection scenarios.

[0004] In a first aspect, the present invention provides a method for correcting in-vehicle screen projection, the method comprising: acquiring projection images of a first screen and a second screen at the same time; inputting the projection images of the first screen and the second screen into a preset classification model respectively, and determining a first state and a second state respectively, wherein the first state is the display state of the projection image of the first screen and the second state is the display state of the projection image of the second screen; comparing the first state and the second state, if they are the same, determining that the projection is normal and continuing to execute the step of acquiring projection images of the first screen and the second screen at the same time respectively; if they are different, determining that the projection is abnormal, and correcting the projection image of the second screen based on the projection image of the first screen.

[0005] Based on the above technical means, the status of the two screens is determined by a classification model. The consistency of the status categories is used as the basis for determining whether the projection is normal. The status of the central control screen is used as a benchmark to correct the second screen, ensuring that the display status of the multiple screens is consistent, so that the driver can obtain synchronized visual information from different line of sight.

[0006] In one optional implementation, if they are different, the projection is determined to be abnormal, and the projection of the second screen is corrected based on the projection of the first screen, including: performing a first-level correction, which is a data synchronization correction; returning to the step of collecting projection images of the first screen and the second screen at the same time, and completing the correction when the first state and the second state become the same; if the first state and the second state are still different, the projection channel data transmission is determined to be abnormal, and a second-level correction is performed, which is a screen synchronization correction.

[0007] Based on the above technical means, the two-level correction complements each other. The first-level correction is for data synchronization to prevent different projection states caused by problems in the data transmission process. The second-level correction serves as a fallback to ensure that anomalies can be resolved in the end, thus achieving a balance between efficiency and reliability.

[0008] In one alternative implementation, performing the first-level correction includes: sending rendering data of the projected image from the first screen to the second screen; and controlling the second screen to redraw the image based on the rendering data.

[0009] Based on the above technical means, the volume of the rendered data is much smaller than that of a complete pixel image, resulting in high transmission efficiency under the condition of limited in-vehicle network bandwidth. The second screen performs local adaptation and rendering of the rendered data according to its own hardware parameters, giving the method good cross-hardware compatibility.

[0010] In one alternative implementation, performing a second-level correction includes: capturing the currently displayed image on the first screen; and transmitting the captured currently displayed image to the second screen for full-screen display.

[0011] Based on the aforementioned technical means, the second screen no longer performs data parsing and rendering operations, but directly uses the captured image as a fallback method to ensure successful correction.

[0012] In one optional implementation, before capturing the projection images of the first screen and the second screen at the same time, the method further includes: capturing multiple frames of first screen screenshots during the operation of the first screen and multiple frames of second screen screenshots during the projection of the second screen; labeling each frame of the first screen screenshot and each frame of the second screen screenshot with a preset classification label to construct an image classification dataset; and training a preset classification model using the image classification dataset.

[0013] Based on the above technical means, the training samples fully cover various screen forms that may occur during actual screen projection. The paired acquisition method enables the model to independently judge the current state of each screen during actual use, providing an accurate basis for state comparison and correction decisions.

[0014] In one optional implementation, each frame of the first screenshot and each frame of the second screenshot are labeled with preset classification labels to construct an image classification dataset, including: selecting screenshots in the first screenshot that contain visual information used to determine a preset state and labeling them with a first preset label; selecting screenshots in the first screenshot that do not contain visual information and labeling them with a second preset label; selecting screenshots in the second screenshot that contain visual information and labeling them with a third preset label corresponding to the first preset label; selecting screenshots in the second screenshot that do not contain visual information and labeling them with a fourth preset label corresponding to the second preset label.

[0015] Based on the above technical means, the labeling systems of the two types of screens are independent but semantically corresponding, enabling the classification model to learn the state features of each of the two screens respectively. The labeling rules are clear and explicit, and have good engineering operability.

[0016] In one optional implementation, the second screen includes at least one of an instrument panel and a HUD, and the step of capturing the projected image of the second screen includes: acquiring the graphics processor rendering output data of the second screen.

[0017] Based on the above technical means, the read rendering output data can be directly used as input samples for the state determination model without the need for camera shooting and image calibration. The data accuracy is high and it does not rely on additional image acquisition hardware, which reduces system cost and complexity.

[0018] Secondly, the present invention provides a vehicle-mounted screen projection device, comprising: a screen projection image acquisition module for acquiring screen projection images of a first screen and a second screen at the same time; a state acquisition module for inputting the screen projection images of the first screen and the second screen into a preset classification model, and determining a first state and a second state respectively, wherein the first state is the display state of the screen projection image of the first screen and the second state is the display state of the screen projection image of the second screen; a state comparison module for comparing the first state and the second state, and if they are the same, determining that the screen projection is normal and continuing to execute the step of acquiring screen projection images of the first screen and the second screen at the same time; and a correction module for determining that the screen projection is abnormal if they are different, and correcting the screen projection image of the second screen based on the screen projection image of the first screen.

[0019] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle projection correction method of the first aspect or any corresponding embodiment described above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the vehicle-mounted screen projection correction method of the first aspect or any corresponding embodiment thereof.

[0021] Fifthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the vehicle projection correction method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the first process of the vehicle-mounted screen projection correction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the second process of the vehicle-mounted screen projection correction method according to an embodiment of the present invention; Figure 3 This is a schematic flowchart of a specific implementation method of the vehicle-mounted screen projection correction method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an in-vehicle projection correction device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] In related technologies, some screen projection solutions improve the display effect of in-vehicle screens by adjusting the target frame rate and target resolution of the image. These solutions focus on optimizing the image quality of a single screen under changes in environmental parameters, but do not address the issue of consistency in the display state between different screens in multi-screen scenarios. When the projection source simultaneously sends image content to multiple screens such as the central control screen and the instrument panel screen, the actual displayed images may be in different functional states due to factors such as hardware differences, different transmission paths, or abnormal data processing. For example, the central control screen may be in navigation mode while the instrument panel screen remains on the non-navigation interface. In this case, although the image quality parameters of each screen have been optimized, the state of the displayed content is inconsistent, and the information obtained by the driver from different line of sight may conflict. Manual intervention from the user is required to restore the state synchronization between the screens. Therefore, how to automatically detect and correct the state differences of each screen in multi-screen projection scenarios and maintain consistency between different screens is a problem that needs further resolution in related technologies.

[0028] According to an embodiment of the present invention, an embodiment of a vehicle-mounted screen projection correction method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a vehicle-mounted screen projection correction method, which can be used in the aforementioned electronic devices. Figure 1 This is a flowchart of the in-vehicle screen projection correction method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Capture the projection images of the first screen and the second screen at the same time.

[0030] Step S102: Input the projection images of the first screen and the second screen into the preset classification model respectively, and determine the first state and the second state respectively. The first state is the display state of the projection image of the first screen, and the second state is the display state of the projection image of the second screen.

[0031] Step S103: Compare the first state and the second state. If they are the same, the screen projection is determined to be normal and the process continues to collect the projection images of the first screen and the second screen at the same time.

[0032] Step S104: If they are different, the screen projection is determined to be abnormal, and the screen projection of the second screen is corrected based on the screen projection of the first screen.

[0033] Specifically, in a vehicle environment, the first screen is the central control screen, and the second screen is another display device besides the central control screen. This display device can be the instrument panel, head-up display, passenger entertainment screen, rear ceiling screen, door panel screen, or armrest screen, etc. As the main display device, the central control screen usually has a more direct and reliable signal path, serving as the reference benchmark. The content displayed on the second screen and the central control screen comes from the same projection source, such as the navigation screen, media playback interface, or vehicle settings panel. To ensure that the driver and passengers receive consistent visual information from different line-of-sight directions, it is necessary to first capture images from the central control screen and the second screen simultaneously. The acquisition action is triggered by a hardware synchronization signal to ensure that the two acquired images have the same timestamp.

[0034] After inputting the screen projection from the central control screen into a preset classification model, the model outputs a first state, which is an evaluation of the display state of the screen projection. The screen projection from the second screen is also input into the preset classification model, which outputs a second state, representing the display state of the second screen projection. The preset classification model is a pre-trained lightweight image processing network used to determine which preset display state category the screen currently belongs to. These preset display state categories are pre-defined based on the functional scenarios of the projection content, such as navigation in progress, no navigation, media playback in progress, vehicle settings in progress, and phone call in progress. The model extracts features from the input screen to identify key elements. While the network architecture of the preset classification models is the same, models deployed on different screens can adapt to different training data and output categories based on their respective functional scenarios. For example, the model output categories for the central control screen cover multiple states such as navigation in progress, no navigation in progress, media playback in progress, vehicle settings in progress, and phone call in progress. However, since the head-up display is primarily used for navigation guidance, its model output categories can only be navigation in progress and no navigation in progress. In practical applications, these categories can be flexibly adjusted according to the functional positioning of different screens; no limitations are imposed here.

[0035] Comparing the first and second states, if they belong to the same state category, the screen projection is considered normal. For example, if the central control screen and the instrument panel display navigation maps simultaneously, the system determines that the two states are the same. At this point, it returns to the image acquisition stage and continues to capture the next pair of images for continuous monitoring.

[0036] If the first and second states differ, the screen projection is deemed abnormal. For example, if the central control screen is currently in navigation mode, while the second screen remains on the non-navigating homepage, the two states are different. The system will then use the central control screen's state as the benchmark to correct the second screen. Specifically, the correction method involves switching the second screen's display content to the same state category as the central control screen, forcing the second screen to refresh its display to show the navigation interface content, thus maintaining consistency with the central control screen. In actual vehicle configurations, the specific type of the second screen can be flexibly selected based on the vehicle model and hardware layout; no restrictions are imposed here.

[0037] The in-vehicle projection correction method provided in this invention uses a classification model to determine the state of the central control screen and the second screen. Both the first and second states represent the functional scene categories to which the images belong. When the states are the same, projection is considered normal and monitoring continues cyclically. When the states differ, the second screen is switched to the same state category based on the state of the central control screen. This state-category-based determination improves the robustness of projection state detection. Furthermore, the central control screen, as the primary display device, is designated as the reference benchmark, resulting in a more direct and reliable signal path. Correcting the second screen based on the central control screen's state ensures consistency between the two screens. This system ensures that drivers and passengers receive consistent visual information from different viewing directions, avoiding cognitive confusion caused by inconsistent display states across multiple screens. This enhances driving safety and passenger experience. The acquisition action is triggered by a hardware synchronization signal, ensuring that the two acquired images have the same timestamp. This eliminates errors caused by software timestamp asynchrony and improves the accuracy of state determination. The closed-loop mechanism of cyclic acquisition and correction followed by re-verification allows the system to track changes in the projection state in real time. Once a difference is detected, it is corrected promptly. After correction, the system re-enters the cyclic verification process to ensure continuous state synchronization. This effectively addresses situations where the projection content changes frequently due to user operations or application switching during driving.

[0038] This embodiment provides a vehicle-mounted screen projection correction method, which can be used in the aforementioned electronic devices. Figure 2 This is a flowchart of the in-vehicle screen projection correction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Capture the projected images of the first screen and the second screen at the same time. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0039] Step S202: Input the projected images from the first screen and the second screen into a preset classification model, and determine the first state and the second state respectively. The first state is the display state of the projected image from the first screen, and the second state is the display state of the projected image from the second screen. For details, please refer to... Figure 1Step S102 of the illustrated embodiment will not be described again here.

[0040] Step S203: Compare the first state and the second state. If they are the same, the screen projection is determined to be normal, and the process continues to capture the projection images of the first screen and the second screen at the same time. For details, please refer to [link to details]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0041] Step S204: If they are different, the screen projection is determined to be abnormal, and the screen projection of the second screen is corrected based on the screen projection of the first screen.

[0042] Specifically, step S204 includes: Step S2041: Perform the first-level correction, which is a data synchronization correction.

[0043] Step S2042: Return to the step of capturing the projection images of the first screen and the second screen at the same time, and complete the correction when the first state and the second state become the same.

[0044] Step S2043: If the first state and the second state are still different, it is determined that the data transmission of the projection channel is abnormal, and the second level of correction is performed. The second level of correction is screen synchronization correction.

[0045] Specifically, the above-mentioned process of correcting the second screen is divided into two levels. First, the first level of correction is performed, which is data synchronization correction. When it is determined that the state of the central control screen and the second screen are different, for example, the central control screen is in the navigation state while the second screen is still on the home page without navigation, the system uses the state of the central control screen as the benchmark to correct the projection data of the second screen so that the state of the second screen is consistent with that of the central control screen. Data synchronization correction is to adjust the projection data of the second screen itself. This can be to adjust the projection data received by the second screen, modify the rendering parameters of the second screen, or reset the output content of the second screen, etc., which are all data-related operations.

[0046] After the first level of correction is completed, return to the step of collecting the projection images of the central control screen and the second screen at the same time. Reacquire the two images and input them into the classification model. Compare the first state and the second state again. If the two become the same, it means that the first level of correction has taken effect and the correction process is over.

[0047] If the two are still not the same, for example, if the second screen still cannot maintain the same state as the central control screen after data synchronization correction, the system determines that there is an abnormality in the data transmission of the projection channel. At this time, the second level of correction is performed, which is screen synchronization correction. Screen synchronization correction is aimed at the problem of the projection channel. When the first level of correction cannot restore the state to consistency, it means that the correction of the original data is no longer possible. Therefore, direct screen synchronization is required at this time. For example, the area image can be directly captured and transmitted, bypassing the data parsing and rendering process on the second screen side.

[0048] The vehicle-mounted screen projection correction method provided in this invention divides the correction process into two levels. First, data synchronization correction is performed and the correction result is verified. Screen synchronization correction is only performed when data synchronization correction fails to achieve state consistency. This avoids screen interruption and response delay caused by immediately resetting the projection channel once a state difference is detected. Furthermore, since the first-level correction directly affects the projection data layer rather than rebuilding the transmission link, it is faster and has less impact on the user's visual experience. The second-level screen synchronization correction serves as a fallback measure, ensuring that screen synchronization can be restored even in extreme situations such as abnormal data transmission. The two levels of correction complement each other, improving both correction efficiency and reliability. After the first-level correction is completed, the screen is re-acquired for state comparison to confirm whether the correction has taken effect. This avoids persistent state inconsistency caused by the system misjudging completion when the correction operation has not actually taken effect. If the states are still different, the second-level correction is performed, making the correction operation more targeted.

[0049] In some optional implementations, step S2041 above includes: Step a1: Send the rendering data of the projected image from the first screen to the second screen.

[0050] Step a2: Control the second screen to redraw the image based on the rendering data.

[0051] Specifically, rendering data is the source data used in the rendering process of the projected screen. It is different from the pixel screen that has already been rendered. When the projection source renders the screen, it generates a set of data to describe the screen content based on the current application scenario. For example, in a navigation scenario, the map tile coordinates, route geometry information, the position and direction of the turn arrows, the content and font attributes of the guide text, or in a media playback scenario, the video stream decoding parameters, audio waveform data, playback progress timestamps, etc. The rendering data is extracted from the rendering pipeline of the central control screen and sent to the second screen through the vehicle network. The rendering data exists in the form of structured information, which makes it easy for the second screen to adapt to the resolution, aspect ratio and color configuration of its own display area.

[0052] After receiving the rendering data from the central control screen, the second screen calls its own graphics rendering engine to regenerate the complete projection screen locally according to the data description. During the rendering process, the second screen adapts the image to its own hardware parameters. For example, if the instrument panel's display area is 1920×720 and has an irregular shape, the rendering engine will draw the received map data and guidance information within the corresponding display area. After the second screen completes the rendering, it outputs the generated image to the display screen, ensuring that the projection content seen by the driver on the second screen is consistent with the content currently displayed on the central control screen in terms of information. Because the second screen uses a local rendering engine, the details of the image it renders may differ slightly due to differences in screen hardware, but the functional information carried by the image remains synchronized with the central control screen.

[0053] In one example, Level 1 correction involves directly and synchronously sending the navigation data (including navigation routes, guidance information, etc.) from the central control screen to the projection terminal (instrument screen or HUD), and controlling the projection terminal to re-render the screen based on the sent navigation data to complete the initial correction. This invention employs a first-level correction data synchronization method to send rendering data from the central control screen to the second screen. The second screen then redraws the image based on this data. In navigation scenarios, the rendering data consists of structured information such as navigation routes, guidance information, and map parameters, rather than a pre-rendered pixel image. This method offers higher transmission efficiency under conditions of limited in-vehicle network bandwidth, enabling a rapid completion of the entire process from data sending to image redrawing, avoiding the high bandwidth consumption and transmission delays caused by directly transmitting pixel images. Furthermore, the second screen performs local adaptation rendering based on its own hardware parameters, ensuring that differences in resolution, aspect ratio, and display area shape between different screens do not affect functionality. The accurate presentation of information ensures that various display devices, such as the instrument panel, HUD, and passenger entertainment screen, can render their respective adapted images locally, giving this correction method excellent compatibility. There is no need to customize differentiated correction strategies for different screens. The first-level correction, as the first-level processing method, directly corrects inconsistencies at the data level. When the projection channel itself is not faulty, the second screen can be restored to the same state category as the central control screen by re-sending the correct rendering data. The correction process consumes low system resources and has little impact on the user's visual experience, avoiding the screen interruption and experience degradation caused by resetting the channel when the data level problem can still be solved.

[0054] In an optional implementation, step S2042 includes: Step b1: Capture the currently displayed image on the first screen.

[0055] Step b2: Transmit the captured currently displayed image to the second screen for full-screen display.

[0056] Specifically, capturing the currently displayed image on the central control screen means directly extracting the rendered image content from the central control screen's display buffer, pixel by pixel. The display buffer stores the complete pixel data of each frame currently being output by the central control screen. The capture operation reads all pixel information of the current frame from the central control screen's frame buffer and generates a bitmap image that is exactly the same as the content displayed on the central control screen. The captured image includes all visual elements currently displayed on the central control screen, such as the complete area of ​​the navigation map, route guidance arrows, text prompts, the vehicle's current location icon, and surrounding road names. These elements have already undergone final color correction, anti-aliasing processing, and layer overlay through the central control screen's rendering pipeline. The captured result is the image that the user is currently viewing on the central control screen.

[0057] The captured current display image is transmitted to the second screen for full-screen display. The system sends the captured bitmap image to the second screen via the vehicle network. After receiving the image, the second screen directly fills the entire display area for output, without performing any additional data parsing, decoding, or rendering. Unlike the method in data synchronization correction where the second screen receives rendering data and draws the image itself, here the second screen only acts as a display. The color value of each pixel on the screen has been pre-determined by the central control screen. The second screen does not need to understand what the content of the image is; it only needs to output the received pixel data point by point to the display screen. For example, in a navigation scenario, the central control screen displays a complete map screenshot, which includes the entire route planning from the current location to the destination. After the system sends this screenshot to the instrument panel, the instrument panel displays the screenshot in full screen. The navigation route seen by the driver on the instrument panel is exactly the same as the route on the central control screen, with no difference in position, color, or markings. Similarly, when the head-up display needs to display turn prompts, the system captures the turn guidance area in the navigation interface of the central control screen and sends the captured small area image to the head-up display for full-screen adaptation. The projected guidance information maintains visual consistency with the guidance information on the central control screen. When the second screen displays the captured image in full screen, it can scale the image according to its own screen aspect ratio and resolution to fit the display area, but does not add or delete any content, ensuring that the displayed content is completely consistent with the image captured by the central control screen.

[0058] In one example, the second-level correction involves taking a real-time screenshot of the current map display on the central control screen and projecting the screenshot directly onto the abnormal instrument panel or HUD. This bypasses the data parsing and rendering process, ensuring that the projected image is completely consistent with the central control screen, thus completely resolving the projection anomaly issue.

[0059] This invention employs a two-stage correction method for image synchronization. The currently rendered image in the central control screen's display buffer is captured as a bitmap and directly transmitted to the second screen for full-screen display. The second screen no longer performs any data parsing, decoding, or rendering; instead, it outputs the received pixel data point by point to the display screen. This completely bypasses the potentially abnormal data parsing and rendering stages in the projection channel. When the first-stage correction fails due to abnormal data processing in the projection channel, the second-stage correction serves as a fallback to ensure the second screen still has content to display. This solves the problem of unrecoverable images caused by decoder failures, rendering pipeline errors, or protocol stack anomalies. The captured image is the complete image currently output by the central control screen, including color correction and anti-aliasing processing. When the image is displayed in full screen on the second screen, all visual elements after being overlaid with the layer are completely consistent with the central control screen at the pixel level. All information such as navigation routes, turn arrows, and text prompts are without deviation, avoiding the detail differences that may occur due to the different rendering of different screens. The content projected on the screen seen by the driver from different viewing directions is exactly the same, eliminating the safety hazards caused by information asymmetry. The secondary correction does not rely on the second screen's ability to analyze the projected data or its local rendering capabilities. Various display devices such as the instrument panel, HUD, and passenger entertainment screen can directly output the received image through the display driver. The hardware performance requirements of the second screen are low. Even if the graphics processor or rendering engine of the second screen malfunctions, the display driver module can still output the received pixel data normally.

[0060] In one optional implementation, prior to step S201, the method further includes: Step c1: Capture multiple frames of screenshots of the first screen during operation, and multiple frames of screenshots of the second screen during projection.

[0061] Step c2: Label each first screenshot and each second screenshot with a preset classification label to construct an image classification dataset.

[0062] Step c3: Train a pre-defined classification model using the image classification dataset.

[0063] Specifically, multiple frames of screenshots of the first screen during operation and multiple frames of screenshots of the second screen during projection are captured. The capture covers various vehicle operating states and user operation scenarios, such as navigation screens when the vehicle is driving on different road conditions like highways and city roads, media playback screens when the vehicle is stationary, and settings interfaces when the user operates in the settings menu. The screenshots of the central control screen and the second screen are captured in pairs, that is, the central control screen and the second screen are captured at the same time to ensure that the captured images reflect the content displayed on the two screens at the same moment.

[0064] Each frame of the first and second screen screenshots is labeled with predefined category labels to construct an image classification dataset. The classification of status categories is based on the functional scenario of the screen content, such as navigation in progress, no navigation, media playback in progress, vehicle settings in progress, and phone call in progress. Annotators observe the content of the images and categorize them according to the functional interface presented. For example, images containing maps, route arrows, and vehicle location icons are labeled as navigation in progress; images containing playback control buttons and progress bars are labeled as media playback in progress; and images containing settings menus and toggle controls are labeled as vehicle settings in progress. Paired images are labeled independently; that is, the central control screen screenshot is labeled with its own corresponding category, and the second screen screenshot is also labeled with its own corresponding category. The labels do not need to be identical between the two, because in actual use, the two screens may display different content. For example, if the central control screen is in navigation mode while the second screen is in a non-navigation state, the central control screen screenshot is labeled as navigation in progress, and the second screen screenshot is labeled as non-navigation in progress. Each is assigned a label that matches its actual displayed content.

[0065] A pre-defined classification model is trained using an image classification dataset. During training, the model learns the correspondence between each image and its labeled category, and learns the ability to classify the input screen into a certain category of the pre-defined state categories. After training, the model can be used to determine the state after real-time screen projection. The current screen of the central control screen and the second screen are input into the model, and the model directly outputs the corresponding state category. The specific network structure and training parameters of the model can be flexibly selected according to the computing power of the vehicle hardware platform, and are not limited here.

[0066] This invention employs pre-collected multi-frame screenshots of the central control screen and the second screen during actual operation, covering various usage scenarios such as vehicle driving, media playback, and settings operations. This ensures that the training samples fully encompass all possible screen display formats during actual projection. The classification model trained based on these samples can accurately identify the screen state under different functional scenarios. The paired acquisition method ensures a one-to-one correspondence between the central control screen screenshot and the second screen screenshot in time. The training data includes the true relationship between the content displayed on the two screens at the same moment, enabling the model to independently determine the current state of each screen during actual use. During the annotation process, the central control screen screenshot and the second screen... Each screenshot is independently labeled, and they are not required to be identical. This fully considers the common situation in real-world in-vehicle environments where the content displayed on multiple screens may differ, such as navigation on the central control screen while navigation is not on the instrument panel. This avoids the distortion of training data or labeling difficulties caused by forcing consistent labels. The classification model obtained after training can automatically output the state category of the real-time captured screen projection without human intervention. The judgment speed is fast and the results are stable, providing an accurate and reliable basis for subsequent state comparison and correction decisions. At the same time, the model structure is lightweight and can be flexibly adjusted according to the computing power of the vehicle hardware, keeping the computing resource consumption of the vehicle's electronic control unit within an acceptable range.

[0067] In one alternative implementation, step c2 includes: Step c21: Select a screenshot of the first screen that contains visual information used to determine the preset state, and label it with the first preset label.

[0068] Step c22: Select a screenshot that does not contain visual information and label it with the second preset label.

[0069] Step c23: Select a screenshot that does not contain visual information and label it with the second preset label.

[0070] Step c24: Select a screenshot containing visual information from the second screen screenshot and label it with a third preset label corresponding to the first preset label.

[0071] Step c25: Select a screenshot of the image that does not contain visual information and label it with the fourth preset label corresponding to the second preset label.

[0072] Specifically, screenshots containing visual information used to determine a preset state are selected from the first screen capture and labeled with a first preset tag. Visual information refers to key elements in the image that indicate the functional scenario of the currently projected content, such as map textures, route arrows, and vehicle location icons specific to navigation interfaces, playback control buttons and progress bars specific to media playback interfaces, or menu lists and switch controls specific to vehicle settings interfaces. The selection rule is to filter the first screen captures, retaining only images that clearly contain the aforementioned visual information, and discarding blurry images, incomplete content, or screenshots where the scenario type cannot be accurately determined. The selected images are assigned a first preset tag, which corresponds to the state category represented by the visual information; for example, a screen in navigation is labeled as "Navigating". Screenshots that do not contain visual information are selected and labeled with a second preset tag. Screenshots that do not contain visual information do not possess any of the key elements mentioned above that can determine the state, such as a black screen, a solid color background, a transition screen during startup, or a system loading animation. Such screens cannot be classified into specific functional scenarios such as "Navigating" or "Media Playback" and are uniformly labeled as "Not Navigating". This tag is the second preset tag.

[0073] For screenshots of the second screen containing visual information, label them with a third preset label corresponding to the first preset label. The third preset label corresponds to the first preset label in terms of state meaning, but applies to the second screen. When comparing states, a corresponding meaning means the same state. For example, when the second screen displays a navigation screen, if it contains the same type of map texture and route arrows as the central control screen, the label on the second screen screenshot is semantically consistent with the label in the central control screen navigation, both indicating the navigation state. However, the label name can be distinguished by the screen source, such as "Instrument Navigation" or "Second Screen Navigation." For screenshots of the second screen that do not contain visual information, label them with a fourth preset label corresponding to the second preset label. Screenshots of the second screen that do not contain visual information, such as the instrument panel's standby black screen or startup loading screen, are labeled as an unknown state corresponding to the second screen. This label corresponds semantically to the second preset label of the central control screen, but belongs to the second screen's category system. In the actual annotation process, screenshots from the first and second screens are processed independently. There is no requirement for a one-to-one correspondence between the labels on the central control screen screenshot and the labels on the second screen screenshot. A central control screen screenshot taken at the same time might be labeled as "navigation in progress," while the second screen screenshot might be labeled as an unknown state with a fourth preset label. The labels for each are determined based on whether and what kind of visual information is present in the image. The specific naming method for the labels can be flexibly chosen according to the actual vehicle configuration and is not limited here.

[0074] In one example, during a navigation scenario, screenshots from map software were collected and categorized into two types: "navigating" and "not navigating." Features marked on the images labeled "navigating" included: route, navigation arrows, remaining distance, next turn indicator, and enlarged view of intersections. Images labeled "not navigating" were marked with: map base map, no route, no arrows, and map screenshots without navigation information.

[0075] Images from the projection devices were collected and categorized into two types: those showing navigation projection and those showing no navigation projection. Images showing navigation projection included features such as turn arrows, lane information, street names, and a distance countdown timer. Images showing no navigation projection only displayed vehicle speed, vehicle logos, and a compass, without navigation guidance.

[0076] This invention, through independent annotation of a first screen screenshot and a second screen screenshot, assigns a first or second preset label to the first screen screenshot and a third or fourth preset label to the second screen screenshot. While the labeling systems for the two screens are independent, they are semantically related. This allows the classification model to learn the state characteristics of each screen during training, and in subsequent practical use, output the corresponding state category for the central control screen and the second screen, providing an accurate data foundation for comparing the states of the two screens. The annotation rules are clear, efficient, and consistent, using the presence of key elements indicating the functional scenario in the image. The third preset label for the second screen corresponds semantically to the first preset label, and the fourth preset label corresponds semantically to the second preset label. When the model outputs the first preset label for the central control screen and the third preset label for the second screen, the meanings are consistent, and the system can determine that the two screens are in the same state. This semantic alignment of the labeling system avoids confusion in state recognition due to different screen sources, enabling cross-screen state comparison based on a unified semantic standard.

[0077] In one optional implementation, the second screen includes at least one of an instrument panel and a HUD, and the step of capturing the projected image from the second screen includes: Step d1: Obtain the graphics processor rendering output data for the second screen.

[0078] Specifically, acquiring the graphics processor rendering output data of the second screen refers to directly reading the image data from the end of the rendering pipeline of the graphics processor (GPU) corresponding to the second screen. When the instrument panel and HUD display projected content, their graphics processors continuously execute rendering tasks, drawing the received navigation data, media data, or vehicle status data into a complete frame. The pixel data of the image is temporarily stored in the frame buffer of the graphics processor in the form of rendering output data before being output to the display screen or projection optical engine. Reading the data directly from this frame buffer yields the precise pixel information of the frame image that the second screen will display. The image contains all the visual elements that the screen should currently present, such as the map area, turn arrows, lane information, road names, and distance countdown on the instrument panel, or the vehicle speed display, navigation guide arrows, and lane markings on the HUD projection screen.

[0079] For instrument cluster displays, the graphics processor's rendering output data is completely consistent with the final image displayed on the LCD panel. This is because instrument clusters are direct display devices; the data in the frame buffer is directly output to the screen's pixel array after passing through the display driver, without any intermediate conversion or distortion. For head-up displays (HUDs), the situation is slightly different. The HUD's graphics processor renders the original image data from the projection light source. This data is amplified and reflected by the optical system to form a virtual image on the windshield. The virtual image itself is affected by the glass curvature and projection distance, but the rendered output data itself is a standard, distortion-free source image. After obtaining the rendered output data from the HUD's graphics processor, it can be directly used as input samples for the state determination model to determine whether the HUD is currently displaying a navigation state or a non-navigation state, without requiring camera capture and image calibration processing.

[0080] The embodiments of the present invention achieve high data accuracy through the above-described acquisition method. The rendered output data is a complete pixel-level image, unaffected by ambient light, shooting angle, or glass reflection. Furthermore, the acquisition action does not rely on additional image acquisition hardware; it can be completed directly by reading from the internal cache of the graphics processor. The read rendered output data can be used for subsequent classification model state determination, or it can serve as a true reflection of the current image on the second screen in the state comparison between the central control screen and the second screen, and can be compared with the image on the central control screen. The specific reading interface and format of the graphics processor's rendered output data can be flexibly selected according to the hardware platform used by the second screen, and are not limited here.

[0081] As a specific embodiment of the present invention, such as Figure 3 As shown, the overall process is divided into three main parts: pre-processing data acquisition and model training, real-time anomaly identification, and screen projection hierarchical correction.

[0082] The initial process involves data collection and model training. Screenshots from the central control screen map software and images from the projection terminal are collected and categorized. Images from the central control screen are categorized into two types: those in navigation mode and those not in navigation mode. Images in navigation mode are labeled with features including route, navigation arrows, remaining distance, next turn prompts, and enlarged intersection views. Images not in navigation mode are only labeled with the map base and do not contain routes, arrows, or any navigation information. Images from the projection terminal are categorized into two types: those in navigation projection mode and those not in navigation projection mode. Images in navigation projection mode are labeled with navigation guidance features such as turn arrows, lane information, road names, and distance countdowns. Images not in navigation projection mode only contain vehicle speed, vehicle icons, and a compass, without any navigation-related guidance content. After data collection and labeling, lightweight image classification networks such as MobileNetV2 / V3, ResNet18, and tinyCNN are used. The labeled dataset is used as training samples to train the model. This model can take a single screen image as input and output the classification result indicating whether the screen is in navigation mode or not.

[0083] After the model training is completed, the system enters the cyclic execution of the anomaly identification step. The system captures the map image of the central control screen in real time and inputs it into the classification model to obtain the central control screen state S1. Simultaneously, it captures the image of the instrument panel or HUD projection terminal and inputs it into the same model to obtain the projection terminal state S2. Then, it compares the states of S1 and S2. When the two states are consistent, the projection is determined to be normal, and the system continues to perform image monitoring and anomaly identification in a loop. When the states of S1 and S2 do not match, the projection is determined to be abnormal, and the system enters the graded correction process.

[0084] The correction process is based on the state of the central control screen, which serves as the initiator of navigation commands. First, a first-level correction is performed, where navigation data containing navigation routes and guidance information from the central control screen is synchronously sent to the projection end. The projection end then re-renders the screen based on this data to complete the initial correction. After the first-level correction, the system re-captures the screen image from the projection end and obtains a new state S2' through the model. S1 and S2' are compared again. If the two states are consistent, the correction is complete, and the system returns to the anomaly identification step for continuous monitoring. If the states still do not match after correction, it is determined that there is a data transmission anomaly in the projection channel, and a second-level correction is performed. The current map image from the central control screen is captured in real time, and the screenshot file is directly projected to the abnormal projection end, bypassing the data parsing and rendering steps, so that the projected image and the central control screen image are completely synchronized. After the second-level correction is completed, the correction is also determined to be complete, and the system returns to the anomaly identification step for continuous loop monitoring, realizing fully automatic identification and fault repair of the in-vehicle multi-screen projection status.

[0085] This embodiment also provides a vehicle-mounted screen projection correction device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0086] This embodiment provides a vehicle-mounted screen projection correction device, such as Figure 4 As shown, it includes: The screen projection acquisition module 401 is used to acquire the projection images of the first screen and the second screen at the same time.

[0087] The state acquisition module 402 is used to input the projection screens of the first screen and the second screen into a preset classification model, and to determine the first state and the second state respectively. The first state is the display state of the projection screen of the first screen, and the second state is the display state of the projection screen of the second screen.

[0088] The state comparison module 403 is used to compare the first state and the second state. If they are the same, the screen projection is determined to be normal and the process of collecting the projection images of the first screen and the second screen at the same time continues.

[0089] The correction module 404 is used to determine the projection abnormality if there is a difference, and to correct the projection of the second screen based on the projection of the first screen.

[0090] The vehicle-mounted screen projection correction device provided in this embodiment of the invention can execute the vehicle-mounted screen projection correction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0091] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0092] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0093] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0094] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the vehicle-mounted screen projection correction method of the embodiments of the present invention.

[0095] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0096] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the in-vehicle projection correction method shown in the above embodiments is implemented.

[0097] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0098] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for correcting in-vehicle projection screens, characterized in that, The method includes: Capture the projected images from the first and second screens at the same time. The projected images of the first screen and the second screen are respectively input into a preset classification model to determine the first state and the second state. The first state is the display state of the projected image of the first screen, and the second state is the display state of the projected image of the second screen. If the first state and the second state are the same, the screen projection is determined to be normal and the step of capturing the projection images of the first screen and the second screen at the same time is continued. If they are different, the screen mirroring is determined to be abnormal, and the screen mirroring of the second screen is corrected based on the screen mirroring of the first screen.

2. The method according to claim 1, characterized in that, If the projection is different, it is determined to be abnormal, and the projection of the second screen is corrected based on the projection of the first screen, including: Perform the first level of correction, which is data synchronization correction; Return to the step of capturing the projection images of the first screen and the second screen at the same moment, and complete the correction when the first state and the second state become the same; If the first state and the second state are still different, it is determined that the data transmission of the projection channel is abnormal, and the second level of correction is performed. The second level of correction is screen synchronization correction.

3. The method according to claim 2, characterized in that, The execution of the first level of correction includes: Send the rendering data of the screen projection from the first screen to the second screen; Control the second screen to redraw the image based on the rendered data.

4. The method according to claim 1, characterized in that, The execution of the second-level correction includes: Capture the currently displayed image of the first screen; The captured currently displayed image is transmitted to the second screen for full-screen display.

5. The method according to claim 1, characterized in that, Before capturing the projection images of the first screen and the second screen at the same moment, the method further includes: Capture multiple frames of screenshots of the first screen during operation, and multiple frames of screenshots of the second screen during projection. Each frame of the first screenshot and each frame of the second screenshot are labeled with a preset classification label to construct an image classification dataset; The preset classification model is trained using the image classification dataset.

6. The method according to claim 5, characterized in that, The step of labeling each frame of the first screen capture and each frame of the second screen capture with preset classification labels to construct an image classification dataset includes: Select a screenshot of the first screen that contains visual information used to determine the preset state, and label it with the first preset label; Select a screenshot that does not contain the aforementioned visual information and label it with the second preset tag; Select a screenshot of the second screen that contains the visual information and label it with a third preset label corresponding to the first preset label; Select a screenshot that does not contain the visual information and label it with a fourth preset label corresponding to the second preset label.

7. The method according to any one of claims 1-6, characterized in that, The second screen includes at least one of an instrument panel and a HUD. The steps for capturing the projected image from the second screen include: Obtain the graphics processor rendering output data for the second screen.

8. A vehicle-mounted screen projection correction device, characterized in that, The device includes: The screen projection capture module is used to capture the projection images of the first screen and the second screen at the same time. The state acquisition module is used to input a preset classification model into the projection screens of the first screen and the second screen respectively, and to determine the first state and the second state respectively. The first state is the display state of the projection screen of the first screen, and the second state is the display state of the projection screen of the second screen. The state comparison module is used to compare the first state and the second state. If they are the same, it is determined that the screen projection is normal and the step of collecting the projection images of the first screen and the second screen at the same time is continued. The correction module is used to determine that the projection is abnormal if there is a difference, and to correct the projection of the second screen based on the projection of the first screen.

9. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the vehicle-mounted screen projection correction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the vehicle-mounted screen projection correction method according to any one of claims 1 to 7.

Images