Electronic rearview mirror image processing method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of intelligent terminals, and provides an electronic rearview mirror image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the real-time fault diagnosis of a camera of an electronic rearview mirror; in response to diagnosis of the first type of fault, executing a first display strategy which comprises the following steps: generating a graphical environment view behind the side of the vehicle based on environment perception data of a non-image sensor, and displaying the graphical environment view on a display screen of an electronic rearview mirror; and in response to diagnosis of the second type of fault, executing a second display strategy, including generating graphical prompt information based on the environment perception data of the non-image sensor, superposing an image collected by the camera and the graphical prompt information, and displaying the superposed image on a display screen of the electronic rearview mirror. When the camera breaks down, it can be ensured that a driver can still obtain key environment information stably and reliably, and therefore the driving risk is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to an electronic rearview mirror image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] As a modern alternative to traditional optical rearview mirrors, electronic rearview mirrors use high-definition cameras deployed on the sides of the vehicle to capture environmental images and display them in real time on the in-vehicle screen, effectively expanding the driver's field of vision and gradually becoming a standard feature of intelligent connected vehicles.

[0003] Current technology typically determines camera malfunctions by monitoring the camera's data stream. When a malfunction is detected, the common practice is to display a black screen or perform a simple reset. This fault handling method cannot recover valid image information when the camera suffers physical damage, the lens is completely obstructed, or it is severely contaminated. This creates a visual interruption and blind spot, causing the driver to completely lose awareness of the environment to the side and rear of the vehicle. In complex traffic environments, this can easily lead to safety accidents and pose a driving hazard.

[0004] Therefore, how to ensure that drivers can still reliably obtain critical environmental information when cameras malfunction, thereby effectively reducing driving risks, is an urgent problem to be solved. Summary of the Invention

[0005] This application provides an electronic rearview mirror image processing method, apparatus, electronic device, and storage medium, which can accurately and effectively evaluate the performance of a large educational model used for classroom question-and-answer pair analysis, and improve the consistency between evaluation results and practical applications.

[0006] In a first aspect, embodiments of this application provide an electronic rearview mirror image processing method, including: Real-time fault diagnosis of the camera in the electronic rearview mirror; In response to the diagnosis of a first type of fault, a first display strategy is executed, the first display strategy including: generating a graphical environmental view of the side and rear of the vehicle based on environmental perception data from a non-image sensor, and displaying the graphical environmental view on the display screen of the electronic rearview mirror; In response to the diagnosis of a second type of fault, a second display strategy is executed, which includes: generating a graphical prompt based on environmental perception data from a non-image sensor, overlaying the image captured by the camera with the graphical prompt, and displaying the overlaid image on the display screen of the electronic rearview mirror.

[0007] In one possible implementation of the first aspect, the real-time fault diagnosis of the camera of the electronic rearview mirror includes: Monitor the interruption signal of the video frame of the camera, or monitor the sequence number count of the image frame of the camera; When an interruption signal of the video frame is detected within a consecutive preset time period, or when the image frame sequence number count stops updating within a consecutive preset time period, the camera is diagnosed as having a first type of fault.

[0008] In one possible implementation of the first aspect, the real-time fault diagnosis of the camera of the electronic rearview mirror includes: Calculate the sharpness of the image output by the camera; When the resolution is lower than a preset resolution threshold, a second type of fault is diagnosed in the camera.

[0009] In one possible implementation of the first aspect, the real-time fault diagnosis of the camera of the electronic rearview mirror includes: The image output by the camera is divided into several image regions, and the average grayscale value of each image region is calculated. When the average grayscale value of an image region differs from the average grayscale value of other image regions in the same frame by more than a preset difference threshold, and this difference persists for several consecutive frames, the camera is diagnosed as having a second type of fault.

[0010] In one possible implementation of the first aspect, generating a graphical environmental view of the vehicle's side and rear based on environmental perception data from a non-image sensor includes: Acquire environmental perception data from radar and / or V2X vehicle-to-everything (V2X) modules; The outline of this vehicle is determined based on its inherent dimensional parameters; Based on the environmental perception data, the outlines of surrounding targets are determined; Based on the outline of the vehicle and the outlines of the surrounding targets, a graphical environment view of the side and rear of the vehicle is generated.

[0011] In one possible implementation of the first aspect, when the environmental perception data comes from radar, the environmental perception data includes target data output by the radar, and determining the contours of surrounding targets based on the environmental perception data includes: Analyze the target data to obtain the original coordinates of surrounding targets in the radar coordinate system; The original coordinates are transformed to the vehicle coordinate system using a pre-calibrated coordinate transformation matrix to obtain the coordinates of the surrounding targets in the vehicle coordinate system. The outline of the surrounding targets is determined based on their coordinates in the vehicle coordinate system.

[0012] In one possible implementation of the first aspect, when the environmental perception data comes from the V2X vehicle-to-everything (V2X) module, the environmental perception data includes the status information of surrounding targets, and the status information includes the latitude and longitude coordinates, heading angle, and vehicle size of the surrounding targets. Determining the contours of surrounding targets based on the environmental perception data includes: Based on the latitude and longitude coordinates of the surrounding targets and the real-time latitude and longitude coordinates of the vehicle, the coordinates of the surrounding targets in the vehicle coordinate system are determined through geodetic coordinate transformation; The outline of the surrounding targets is determined based on the coordinates of the surrounding targets in the vehicle coordinate system, the heading angle, and the vehicle dimensions.

[0013] In one possible implementation of the first aspect, the overlaying of the image captured by the camera with the graphical prompt information includes: The perspective projection model, established by pre-calibrating the internal and external parameters of the camera, maps the position of the target perceived by the non-image sensor in the vehicle coordinate system to the corresponding pixel coordinates in the image captured by the camera. At the position corresponding to the pixel coordinates, the graphical prompt information is overlaid on the image as a graphical layer.

[0014] Secondly, embodiments of this application provide an electronic rearview mirror image processing device, including: The fault diagnosis unit is used to perform real-time fault diagnosis on the camera of the electronic rearview mirror. The first fault processing unit is configured to execute a first display strategy in response to the diagnosis of a first type of fault. The first display strategy includes: generating a graphical environmental view of the side and rear of the vehicle based on environmental perception data from a non-image sensor, and displaying the graphical environmental view on the display screen of the electronic rearview mirror. The second fault processing unit is used to execute a second display strategy in response to the diagnosis of a second type of fault. The second display strategy includes: generating graphical prompt information based on environmental perception data from a non-image sensor, overlaying the image captured by the camera with the graphical prompt information, and displaying the overlaid image on the display screen of the electronic rearview mirror.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the electronic rearview mirror image processing method as described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the electronic rearview mirror image processing method as described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the electronic rearview mirror image processing method as described in the first aspect above.

[0018] In this embodiment, real-time fault diagnosis of the camera accurately identifies the fault type. When a first-type fault is diagnosed, a graphical environmental view of the vehicle's side and rear is generated based on environmental perception data from a non-image sensor. This view completely replaces the failed camera image on the display screen, providing the driver with an uninterrupted, graphical, direct view and avoiding safety hazards caused by complete loss of vision. When a second-type fault is diagnosed, graphical prompts are generated based on environmental perception data from a non-image sensor. The image captured by the camera is superimposed on this graphical prompt, and the superimposed image is displayed on the screen. By superimposing graphical prompts on the camera image, potential distortions or incompleteness in the fault image can be accurately compensated for, effectively enhancing the driver's ability to identify key environmental information in the image. This solution implements differentiated display strategies for different fault categories, ensuring that the driver can consistently and reliably obtain key environmental information from the electronic rearview mirror display screen when the camera malfunctions, thereby effectively reducing driving risks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the implementation of the electronic rearview mirror image processing method provided in this application embodiment; Figure 2 This is a flowchart illustrating a specific implementation of the electronic rearview mirror image processing method for diagnosing a first type of fault, as provided in this application embodiment. Figure 3.1 This is a flowchart illustrating a specific implementation of the electronic rearview mirror image processing method for diagnosing a second type of fault, as provided in this application embodiment. Figure 3.2 This is another specific implementation flowchart of the second type of fault in the electronic rearview mirror image processing method provided in the embodiments of this application; Figure 4 This is a flowchart illustrating a specific implementation of the first display strategy in the electronic rearview mirror image processing method provided in this application embodiment; Figure 4.1 This is a flowchart illustrating a specific implementation of the electronic rearview mirror image processing method provided in this application for generating a graphical environment view; Figure 5 This is a flowchart illustrating a specific implementation of the second display strategy for the electronic rearview mirror image processing method provided in this application embodiment; Figure 6 This is a flowchart illustrating a specific implementation of configuring prompt attributes in the electronic rearview mirror image processing method provided in this application embodiment; Figure 7 This is a structural block diagram of the electronic rearview mirror image processing device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] By way of example and not limitation, the electronic rearview mirror image processing method provided in this application is applicable to various types of electronic devices that require electronic rearview mirror image processing. The electronic devices mainly include devices integrated with vehicles, such as the electronic rearview mirror itself or a smart vehicle terminal; in addition, it can also be applied to mobile devices that communicate with the vehicle, such as mobile phones, tablets, wearable devices, etc., for users to remotely view or interact with. This application does not impose any limitations on the specific type of electronic device.

[0028] The electronic rearview mirror image processing method provided in this application is mainly applicable to vehicle electronic rearview mirror systems. Those skilled in the art, based on their understanding of the principles of this invention, can apply it to other in-vehicle vision systems that require providing a rear view, such as electronic interior rearview mirror systems.

[0029] Figure 1 The implementation flow of the electronic rearview mirror image processing method provided in this application embodiment is illustrated. The method flow includes steps S101 to S103. The specific implementation principle of each step is as follows: Step S101: Perform real-time fault diagnosis on the camera of the electronic rearview mirror.

[0030] The camera in an electronic rearview mirror is a sensor mounted on the side of the vehicle to collect images of the environment behind and to the side of the vehicle. The effectiveness of the environmental information provided depends on the camera's operational status.

[0031] Fault diagnosis is the process of monitoring and assessing the operational status of a camera. It involves identifying whether a camera is malfunctioning, distinguishing the type of malfunction, and providing a basis for subsequent differentiated processing decisions. The scope of fault diagnosis covers the entire functional state of the camera, from data output to image quality.

[0032] In this embodiment, fault diagnosis is performed continuously and uninterruptedly at specific intervals during vehicle operation, aiming to achieve timely fault detection and meet the system response speed requirements for driving safety. In one possible implementation, the specific interval can be determined based on the frame rate of the camera's output images. By performing real-time fault diagnosis on the electronic rearview mirror's camera, camera anomalies can be proactively and promptly detected.

[0033] For example, the fault diagnosis and the image acquisition process of the camera are synchronized and run in parallel. The time interval between the diagnostic operations does not exceed the single frame acquisition cycle of the camera. For example, when the camera frame rate is 30fps, the diagnostic frequency is not less than 30 times / second, ensuring that the fault can be captured in time after it occurs, and avoiding the continuous output of invalid images in the fault state due to the delay in fault diagnosis, which would affect driving safety.

[0034] In this embodiment, the camera exhibits two types of faults. The first type is a fault caused by hardware failure (such as power outage, sensor damage, or broken data transmission lines), resulting in an inability to stably output a continuous data stream. Under this type of fault, the camera cannot provide any valid image data. The second type of fault occurs when the camera's basic hardware functions (power supply, sensor light sensing, data transmission) are normal and can stably output a complete image, but external environmental interference (such as lens obstruction, direct sunlight, or heavy rain blur) or camera performance degradation (such as abnormal exposure due to aging of the photosensitive element) prevents the image content from providing the driver with effective environmental information. Unlike the first type of fault, which involves complete hardware failure, the first type of fault involves image output but with poor or invalid information.

[0035] In one possible implementation, the continuity of the camera's data stream is monitored. If a discontinuity in the data stream is detected within a preset time period, a first-type fault in the camera is diagnosed. Monitoring the continuity of the camera's data stream involves performing a low-level communication layer continuity check on the data stream output by the camera. The monitoring object is not the image itself, but rather the communication signal carrying the image data.

[0036] As one possible implementation of this application Figure 2 A specific implementation flow of step S101 in the electronic rearview mirror image processing method provided in this application embodiment is shown below: A1: Monitor the interruption signal of the video frame of the camera, or monitor the sequence number count of the image frame of the camera.

[0037] A2: When an interruption signal of the video frame is detected within a continuous preset time period, or when the image frame sequence number is detected to stop updating within a continuous preset time period, the camera is diagnosed as having a first type of fault.

[0038] The continuous preset time period is a calibrable time window, for example, five consecutive cycles, each cycle being 100ms. Setting the continuous preset time period can filter out instantaneous vibrations in the vehicle environment, avoiding misdiagnosis. Based on dual verification of time continuity and data stream continuity, the system diagnoses whether the camera has a Type I fault, thereby improving the accuracy of fault diagnosis.

[0039] In one possible implementation, the continuity of the camera's data stream is determined by capturing interruption signals of camera video frames. When an interruption signal is captured within a preset time period, the data stream of the camera is determined to be discontinuous. This embodiment verifies data stream continuity by monitoring changes in interruption signals at the hardware level.

[0040] For example, with a detection cycle of 100 milliseconds and a camera frame rate of 60 frames per second, it is expected that 6 interrupt signals should be generated within each 100-millisecond cycle, and the corresponding interrupt counter should increment by 6. At the end of each detection cycle, the change in the value of the interrupt counter is checked. For example, at the initial time t0, the recorded interrupt counter value is N. After the first 100-millisecond cycle (time t1), the interrupt counter value is read again. If its value is still N and the expected increase has not occurred, then the data flow is recorded as abnormal in this cycle, and the verification process continues. If the interrupt counter value stops incrementing and remains at N for 5 consecutive cycles (i.e., a preset time period of 500 milliseconds), then it is determined that the camera's data flow is discontinuous, that is, the camera is diagnosed with a first-type fault.

[0041] In one possible implementation, the continuity of the camera's data stream is determined by monitoring whether the frame sequence number count stops updating. When the frame sequence number count stops updating within a preset time period, the data stream is determined to be discontinuous. This embodiment performs data stream continuity verification by monitoring the frame sequence number of the data protocol at the application layer.

[0042] For example, a detection cycle is set at 100 milliseconds, the camera frame rate is 60 frames per second, and each frame contains a monotonically increasing frame sequence number. Within each detection cycle, the received data packets are parsed and the current frame sequence number is recorded. For instance, at the initial time t0, the recorded frame sequence number is S. After the first 100-millisecond cycle (time t1), the received data packets are parsed. If the frame sequence number is still S and the expected increase has not occurred, an abnormal data flow is recorded for that cycle, and the monitoring process continues. If, within five consecutive cycles (i.e., a preset time period of 500 milliseconds), the frame sequence number in all received data packets stops increasing and remains at S, then it is determined that the camera's data flow is discontinuous, thus diagnosing a type I fault in the camera.

[0043] In one possible implementation, a validity analysis is performed on the images output by the camera to diagnose whether the camera has a type II fault. Validity analysis refers to the process of calculating and evaluating image data that has been successfully output by the camera but whose quality or integrity is compromised. In this embodiment, the validity analysis does not focus on the specific content of the image, but rather assesses its usability as a carrier of environmental information.

[0044] As one possible implementation of this application, the validity analysis includes sharpness analysis. Figure 3.1 This application provides a specific implementation flow for diagnosing whether a camera has a second type of fault in its electronic rearview mirror image processing method, as detailed below: B1: Calculate the sharpness of the image output by the camera. Sharpness refers to the sharpness of details and edges in an image, used to evaluate whether an image is blurry. In in-vehicle scenarios, drivers need to identify key information such as lane lines, obstacle edges, and vehicle outlines through electronic rearview mirror images; sharpness directly determines the recognizability of this key information.

[0045] B2: When the resolution is lower than a preset resolution threshold, a second type of fault is diagnosed in the camera.

[0046] A preset sharpness threshold is used to define the boundary between sharpness and blurriness. When the calculated sharpness value is consistently lower than the preset threshold, it indicates that the overall image output by the camera is blurry and can no longer provide the driver with effective detail information. This is considered a type II camera malfunction.

[0047] In one possible implementation, a second type of fault is diagnosed in the camera only when the clarity of several consecutive frames (e.g., three consecutive frames) is lower than a preset clarity threshold. If the clarity of only one or two frames is lower than the preset clarity threshold but subsequent frames recover, it is determined to be transient interference and is not determined to be a second type of fault.

[0048] One possible implementation uses the Laplacian variance method to calculate the sharpness of the image output by the camera. The image is converted to grayscale, and a convolution operation is performed using the Laplacian operator to calculate the second derivative of the image. Edges and detail areas in the image are highlighted, and finally, the variance of the entire Laplacian response image is calculated. The larger the variance value, the richer the details and edges in the image, and the sharper the image.

[0049] For example, the Laplace kernel is a 3×3 matrix: The convolution operation is as follows (1): (1) Where * denotes convolution operation, Let represent the Laplacian operator (second-order differential operator), which is often represented as a predefined convolution kernel in the discrete domain. I(x, y) represents the gray value of the image captured by the camera at pixel coordinates (x, y). The variance is calculated according to the following equation (2): (2) Where N is the total number of pixels in the image. The mean and variance of the Laplace response. The larger the value, the clearer the image. A preset clarity threshold of 100 is set. If Var(L) < 100, the image is considered blurry, and a second type of camera malfunction is diagnosed.

[0050] By calculating the Laplacian variance of each frame in real time, the degree of blur can be quickly and accurately quantified. Once the calculated variance value is consistently lower than the preset sharpness threshold, the second type of fault caused by lens contamination can be automatically diagnosed without manual intervention.

[0051] As one possible implementation of this application, the validity analysis includes integrity analysis. Figure 3.2 The following is a detailed description of another specific implementation flow of the electronic rearview mirror image processing method provided in this application for diagnosing whether the camera has a second type of fault: C1: Divide the image output by the camera into several image regions and calculate the average grayscale value of each image region.

[0052] Dividing an image into several image regions refers to spatially segmenting a complete image frame into multiple independent analysis units. For example, an image can be divided into four equally sized image regions—upper left, upper right, lower left, and lower right—to allow for independent evaluation of local image characteristics. For each segmented image region, the grayscale values ​​of all pixels within that region are iterated, summed, and then divided by the total number of pixels in the region to obtain the average grayscale value of that region. This average grayscale value reflects the overall brightness level of the image region and can be used to identify local brightness anomalies caused by occlusion.

[0053] In one possible implementation, the image segmentation is not random, but based on the field of view function requirements of the vehicle electronic rearview mirror, adopting the principle of prioritizing key areas, the image is divided into functional image areas such as the vehicle side and rear close-range warning area, lane line recognition area, and long-range field of view area, which covers the entire field of view while highlighting the core key areas.

[0054] C2: When the grayscale mean of an image region differs from the grayscale mean of other image regions in the same frame by more than a preset difference threshold, and this difference persists for several consecutive frames, the camera is diagnosed as having a second type of fault.

[0055] The existence of a difference refers to a situation where the average grayscale value of a certain region in a single frame image deviates significantly from the average grayscale value of other regions, indicating that there may be abnormal occlusion in that image region. However, to avoid misjudgment caused by single-frame jitter or instantaneous light and shadow changes, when the average grayscale value of this region in several consecutive frames of images continuously differs from the average grayscale value of other image regions by more than a preset difference threshold (such as 30%), the camera is diagnosed as having a second type of fault.

[0056] In this embodiment, by dividing the image region and continuously calculating the gray-scale mean of each region, the presence of local occlusion regions in the image is detected. By judging the stability of the difference in gray-scale mean in consecutive frames, the second type of fault caused by local occlusion can be accurately diagnosed, without misjudgment caused by interference such as the shadow of a vehicle passing by in an instant.

[0057] For example, each frame of image is divided into 4 equal regions (top left, top right, bottom left, and bottom right), and the grayscale mean of each region is calculated. If the difference between the grayscale mean of a certain region and the grayscale mean of other regions exceeds 30% in multiple consecutive frames, it is determined to be a local occlusion, and the camera is diagnosed to have a second type of fault.

[0058] In one possible implementation, the histogram variance of the grayscale image output by the camera is calculated; when the histogram variance of most frames in a series of consecutive images is less than a preset variance threshold, the camera is diagnosed as having a type 1 fault due to global occlusion.

[0059] For example, the image is divided into 256 bins, and the grayscale histogram is calculated. If the histogram variance of 8 out of 10 consecutive frames is less than a preset variance threshold, it is determined to be a global occlusion. At this time, the camera is diagnosed to have a first type of fault.

[0060] Step S102: In response to the diagnosis of a first type of fault, execute the first display strategy.

[0061] In this embodiment, when a first type of fault is diagnosed, a first display strategy is executed. The first display strategy is a solution for scenarios where the camera is completely malfunctioning (including situations where the camera is completely blocked).

[0062] As one possible implementation of this application Figure 4 The following is a detailed description of a specific implementation flow of the first display strategy in the electronic rearview mirror image processing method provided in this application embodiment: Step S1021: Generate a graphical environmental view of the side and rear of the vehicle based on environmental perception data from non-image sensors.

[0063] Non-image sensors refer to sensing devices that do not rely on optical imaging principles. The environmental perception data output by non-image sensors is not a pixel matrix, but rather structured data containing information such as target distance, azimuth angle, relative speed, and target size. In this embodiment, the non-image sensor includes radar and / or a V2X vehicle-to-everything (V2X) module. The graphical environment view is a dynamically updated, vectorized schematic diagram of the scene centered on the vehicle.

[0064] In one possible implementation, the visual scale of the graphical environment view can be dynamically adjusted according to the resolution of the electronic rearview mirror display to optimize the display effect.

[0065] As one possible implementation of this application Figure 4.1 This application illustrates a specific implementation flow of generating a graphical environment view in the electronic rearview mirror image processing method provided in the embodiment of the present application, which is described in detail below: D1: Acquire environmental perception data from radar and / or V2X vehicle-to-everything (V2X) modules.

[0066] The radar can be millimeter-wave radar or lidar, etc. The environmental perception data of the radar includes, but is not limited to, parameters such as the distance, azimuth, and relative speed of targets around the vehicle.

[0067] The environmental perception data of the V2X vehicle-to-everything (V2X) module includes the status information of surrounding targets (such as other vehicles) broadcast by V2X communication, specifically including but not limited to data such as latitude and longitude coordinates, speed, and heading angle.

[0068] D2: Determine the outline of the vehicle based on its inherent dimensional parameters.

[0069] The vehicle outline is a standard geometric shape (usually a rectangle or a simplified vehicle polygon) pre-constructed based on the vehicle's inherent dimensional parameters (such as vehicle length, width, and wheelbase) as defined by the manufacturer.

[0070] In one possible implementation, after the vehicle outline is determined, the attitude (i.e., heading angle) of the vehicle outline can be dynamically corrected in real time before display by integrating data such as steering angle and yaw angle from the vehicle body attitude sensor, thereby ensuring that it can truly reflect the current driving direction and attitude of the vehicle in the graphical environment view, rather than a fixed direction.

[0071] D3: Based on the environmental perception data, determine the outline of surrounding targets.

[0072] The outlines of surrounding targets are generated based on the fitting of environmental perception data.

[0073] In this embodiment of the application, before determining the outline of the surrounding targets, the acquired environmental perception data is converted to a unified coordinate system. After passing through the coordinate system, the position and distance of each surrounding target relative to the vehicle can be determined based on the environmental perception data, and then the outline of the target is represented by a preset geometric shape (such as a rectangle reflecting the aspect ratio of the target).

[0074] In one possible implementation, when the environmental perception data comes from radar, the environmental perception data includes target data output by the radar. The target data is parsed to obtain the original coordinates of surrounding targets in the radar coordinate system. These original coordinates originate from the processing of the original radar echo signal and directly reflect the target's azimuth and distance relative to the radar sensor itself. Using a pre-calibrated coordinate transformation matrix, the original coordinates are transformed to the vehicle coordinate system to obtain the coordinates of the surrounding targets in the vehicle coordinate system. Based on the coordinates of the surrounding targets in the vehicle coordinate system, the outline of the surrounding targets is determined.

[0075] Specifically, for environmental perception data from radar, a coordinate transformation matrix is ​​established using pre-calibrated installation offsets. These installation offsets characterize the deviations of the radar sensor's physical installation position (translation) and angle (rotation) on the vehicle relative to the origin of the vehicle coordinate system. Based on this coordinate transformation matrix, the target coordinates in the radar sensor coordinate system are transformed to a unified vehicle coordinate system, thus obtaining the coordinates of surrounding targets in the vehicle coordinate system. Then, based on these coordinates, the contour position and orientation of the surrounding targets in the unified vehicle coordinate system can be determined.

[0076] For example, the installation offset is (Δx, Δy, Δθ), where (Δx, y) is the translation and Δθ is the rotation. The coordinate transformation matrix includes: X veh =X sensor ×cosΔθ-Y sensor ×sinΔθ+Δx;Y veh =X sensor ×sinΔθ+Y sensor ×cosΔθ+Δy. X veh Y veh This represents the target coordinates in the vehicle coordinate system, X. sensor Y sensor Represents the original coordinates in the radar sensor coordinate system, based on X. veh Y veh This determines the target's outline position and orientation in the vehicle coordinate system.

[0077] In one possible implementation, when the environmental perception data comes from a V2X vehicle-to-everything (V2X) module, the environmental perception data includes the status information of surrounding targets, where the surrounding targets are target vehicles. This status information is generated and actively broadcast by the target vehicle itself, and includes the latitude and longitude coordinates, heading angle, and vehicle size of the surrounding targets. Based on the latitude and longitude coordinates of the surrounding targets and the real-time latitude and longitude coordinates of the vehicle, the coordinates of the surrounding targets in the vehicle coordinate system are determined through geodetic coordinate transformation. This transformation process converts the geodetic coordinates of the Global Positioning System (WGS84) into relative position coordinates with the vehicle as a reference. Based on the coordinates of the surrounding targets in the vehicle coordinate system, the heading angle, and the vehicle size, the outline of the surrounding targets is determined. This outline includes not only position but also attitude and actual physical size.

[0078] For example, environmental perception data from the V2X vehicle-to-everything (V2X) module is converted from the WGS84 geodetic coordinate system (longitude: Lon, latitude: Lat) to the vehicle coordinate system (e.g., with the rear axle center of the vehicle as the origin, the X-axis pointing forward and the Y-axis pointing left). The conversion formula is: X local =(Lon-Lon veh )×C×cos(Lat veh ×π / 180); Y local =(Lat-Lat veh )×C, where C is the Earth's circumference conversion factor, X loca Y local Lon and Lat represent the coordinates of the target in the vehicle coordinate system, respectively. Lon and Lat represent the latitude and longitude coordinates of the surrounding target obtained by V2X communication. veh Lat veh This indicates the real-time latitude and longitude coordinates of the vehicle provided by the onboard GNSS module, based on X. loca Y local By combining the heading angle and the vehicle dimensions of surrounding targets, the outline of surrounding targets is determined.

[0079] D4: Generate a graphical environment view of the vehicle's side and rear based on the vehicle's outline and the outlines of the surrounding targets.

[0080] Using computer graphics rendering technology, the determined outlines of the vehicle and surrounding targets are laid out, drawn, and composited in a specific view (such as a top view or a side-rear view) to obtain a graphical environmental view of the vehicle's side and rear.

[0081] In this embodiment, when the camera fails completely, environmental perception data is acquired through non-image sensors such as radar and V2X vehicle networking modules. Based on this environmental perception data, the outline of the vehicle and surrounding targets is effectively determined, thereby generating a graphical environmental view of the side and rear of the vehicle. This process converts abstract environmental perception data into an intuitive graphical display that reflects real-time environmental relationships, thus enabling the driver to still be provided with stable and reliable environmental situational awareness even after the camera fails completely.

[0082] Step S1022: Display the graphical environment view on the screen of the electronic rearview mirror.

[0083] In this embodiment, the first display strategy abandons the reliance on the faulty camera and instead uses environmental perception data from non-image sensors to dynamically generate a new, graphical view of the vehicle's side and rear environment. This graphical view is then displayed on the screen of the electronic rearview mirror to completely replace the original camera image, ensuring that the driver can still be provided with stable and reliable key environmental information even when the camera is completely faulty.

[0084] For example, upon diagnosing a Type I fault in the camera due to hardware damage, the system immediately responds and executes a first display strategy. At this point, the system no longer attempts to recover or wait for the camera image. Instead, based on radar detection data, it acquires the distance and azimuth data and, through a graphics rendering engine, generates a graphical environment view on the display screen centered on the vehicle's image and including the outlines of surrounding vehicles. Although this graphical environment view is not a true optical image, it still provides the driver with continuous and reliable information on the location and distribution of key obstacles, thereby ensuring the continuity of environmental perception.

[0085] Step S103: In response to the diagnosis of a second type of fault, execute the second display strategy.

[0086] In this embodiment, when a second type of fault is diagnosed, a second display strategy is executed. The second display strategy is a solution for scenarios where camera performance is degraded (partial occlusion or poor image quality). In the second display strategy, the remaining, partially valid camera images are retained as the display base.

[0087] As one possible implementation of this application Figure 5 The following is a detailed description of a specific implementation flow of the first display strategy in the electronic rearview mirror image processing method provided in this application embodiment: Step S1031: Generate graphical prompts based on environmental perception data from non-image sensors.

[0088] Graphical cues are visual elements superimposed on images, designed to aid in understanding their content.

[0089] In one possible implementation, based on the environmental perception data, one or more graphical prompts are generated, including surrounding target outlines, distance text, and risk level indicators. The surrounding target outlines are used to delineate and highlight objects in the image; the distance text is used for annotation; the relative distance between the target and the vehicle is indicated; and the risk level indicator is used to identify the risk level of the target. The graphical prompts originate from the processing of environmental perception data from non-image sensors such as radar and V2X vehicle-to-everything (V2X) modules. By extracting missing or blurred key parameters from the environmental perception data, these parameters are converted into intuitive graphical prompts.

[0090] Step S1032: Overlay the image captured by the camera with the graphical prompt information, and display the overlaid image on the display screen of the electronic rearview mirror.

[0091] In one possible implementation, overlaying the image captured by the camera with the graphical prompt information means using image processing technology to overlay the graphical prompt information as an independent graphic layer onto the original image layer of the image captured by the camera. This overlay process is based on precise synthesis using coordinate mapping.

[0092] In one possible implementation, a perspective projection model is established using pre-calibrated camera intrinsic and extrinsic parameters to accurately map the position of the target perceived by the non-image sensor in the vehicle coordinate system to its corresponding pixel coordinates in the camera image. At the position corresponding to the pixel coordinates, the graphical prompt information is superimposed on the image as a graphical layer, thereby ensuring that the graphical prompt information such as the target outline can accurately cover the position of the corresponding target in the image.

[0093] In one possible implementation, the pixel coordinates of the target on the camera imaging plane are calculated using a pre-calibrated perspective projection model, specifically according to the following formula (3): (3) Where u and v represent the pixel coordinates of the target on the camera's virtual imaging plane. This represents the camera intrinsic parameter matrix, which includes the camera focal length (f). x f y The principal point coordinates are (u0, v0); and Let X be the camera extrinsic parameter matrix, representing the rotation and translation transformation from the vehicle coordinate system to the camera coordinate system. This matrix includes the camera's installation pitch angle (Pitchα), yaw angle (Yawγ), roll angle (Rollβ), and installation position (Δx, Δy, Δz). f represents the standard camera perspective projection calculation process, and X... local Y local Zground The X represents the target's coordinates in the vehicle's coordinate system. local This represents the target's coordinates in the vehicle's coordinate system along the vehicle's direction of travel (X-axis), and Y-axis... local : Represents the target's coordinates in the vehicle's coordinate system, specifically the coordinates along the vehicle's transverse direction (Y-axis), Z. ground This represents the target's coordinates in the vehicle's coordinate system, along the direction perpendicular to the ground (Z-axis). In this embodiment, Z... ground It is a fixed value, namely Z ground =0. This mapping process ensures that graphical prompts, such as the target outline, accurately cover the corresponding target location in the image.

[0094] In this embodiment, the second display strategy, while retaining the real-time image output by the faulty camera, overlays graphical prompts generated by environmental perception data from non-image sensors. By overlaying the graphical prompts onto the real-time image, the overlaid image is displayed on the screen of the electronic rearview mirror, thereby enhancing the information recognizability of the original image and ensuring that accurate and reliable key environmental information can still be provided to the driver even when the camera image quality deteriorates.

[0095] As one possible implementation of this application Figure 6 This paper illustrates a specific implementation flow of configuring prompt attributes in the electronic rearview mirror image processing method provided in an embodiment of this application, detailed below: E1: Based on the environmental perception data from the non-image sensor, obtain the relative distance and relative speed between the surrounding targets and the vehicle.

[0096] Relative distance refers to the spatial interval between the target and the vehicle; relative speed refers to the rate at which the target approaches or moves away from the vehicle along the line connecting them. Relative distance and relative speed can be derived from environmental perception data from radar or V2X vehicle-to-everything (V2X) modules and serve as the primary basis for objective risk quantification.

[0097] E2: Determine the risk level of the surrounding targets based on the relative distance and the relative speed.

[0098] In one possible implementation, the risk level is divided into three levels: high-risk, medium-risk, and low-risk. This embodiment converts abstract risk into identifiable risk levels. When the relative distance is less than a first distance threshold and the relative speed is greater than a first speed threshold, it is determined to be a high-risk level; when the relative distance is between the first distance threshold and a second distance threshold, or when the relative speed is between the second speed threshold and the first speed threshold, it is determined to be a medium-risk level; when the relative distance is greater than or equal to the second distance threshold and the relative speed is less than the second speed threshold, it is determined to be a low-risk level; wherein, the second distance threshold is greater than the first distance threshold, and the second speed threshold is less than the first speed threshold.

[0099] For example, when the relative distance is <5m and the relative speed is >10m / s, it is judged as "high risk"; when 5m≤relative distance<15m or 5m / s≤relative speed≤10m / s, it is judged as "medium risk"; when the relative distance is ≥15m and the relative speed is <5m / s, it is judged as "low risk".

[0100] In one possible implementation, the risk level is updated in real time according to the target's motion state, and the update frequency is synchronized with the data acquisition of non-image sensors. When the relative speed or distance of the target changes and causes the risk value to jump to a higher level, the level update is triggered immediately.

[0101] E3: Configure different prompt attributes for the graphical environment view or the graphical prompt information according to the risk level.

[0102] Configuring different prompt attributes refers to mapping a determined risk level to a specific, differentiated visual presentation scheme. In this embodiment, different prompt attributes are configured for the outline of the target in the graphical environment view or the graphical prompt information, based on the risk level.

[0103] In one possible implementation, the alert attribute includes a visual attribute. Red is configured for high-risk levels, yellow for medium-risk levels, and green for low-risk levels. This alert attribute configuration is applied to the target outline in the graphical environment view, or the target outline box in the graphical alert information, thereby achieving color coding for different risk targets and more intuitively displaying the risk level of the identified target.

[0104] In one possible implementation, the alert attributes include visual and auditory attributes. While configuring different colors as visual attributes, different frequencies of beeping alerts are triggered based on the risk level: red and continuous high-frequency beeping for high-risk levels, yellow and intermittent medium-frequency beeping for medium-risk levels, and green and a single low-frequency beeping or no beeping for low-risk levels. This multimodal alert attribute configuration is applied to the target outline in the graphical environment view, or the target outline frame in the graphical alert information, thereby achieving differentiated warnings for different risk targets through both visual and auditory channels, more effectively improving the driver's hazard perception and response speed.

[0105] In this embodiment of the application, environmental perception data from non-image sensors is converted into differentiated visual and auditory information that drivers can quickly perceive, which can significantly reduce driving risks caused by blurred visual information.

[0106] As can be seen from the above, in this embodiment, by performing real-time fault diagnosis on the camera, the fault type is accurately identified. When a first type of fault is diagnosed, a graphical environmental view of the vehicle's side and rear is generated based on environmental perception data from non-image sensors. This view completely replaces the failed camera image on the display screen, providing the driver with an uninterrupted, graphical, direct view and avoiding safety hazards caused by complete loss of vision. When a second type of fault is diagnosed, graphical prompts are generated based on environmental perception data from non-image sensors. The image captured by the camera is superimposed on this graphical prompt, and the superimposed image is displayed on the screen. By superimposing graphical prompts on the camera image, the distortion and incompleteness of the fault image can be accurately compensated for, effectively enhancing the driver's ability to identify key environmental information in the image. This application implements differentiated display strategies for different fault categories, ensuring that the driver can consistently and reliably obtain key environmental information from the electronic rearview mirror display screen when the camera malfunctions, thereby effectively reducing driving risks. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] Corresponding to the electronic rearview mirror image processing method described in the above embodiments, Figure 7 A structural block diagram of the electronic rearview mirror image processing device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0108] Reference Figure 7 The electronic rearview mirror image processing device includes: a fault diagnosis unit 71, a first fault processing unit 72, and a second fault processing unit 73, wherein: The fault diagnosis unit 71 is used to perform real-time fault diagnosis on the camera of the electronic rearview mirror. The first fault processing unit 72 is configured to execute a first display strategy in response to the diagnosis of a first type of fault. The first display strategy includes: generating a graphical environmental view of the side and rear of the vehicle based on environmental perception data from a non-image sensor, and displaying the graphical environmental view on the display screen of the electronic rearview mirror. The second fault processing unit 73 is used to execute a second display strategy in response to the diagnosis of a second type of fault. The second display strategy includes: generating graphical prompt information based on environmental perception data from a non-image sensor, superimposing the image captured by the camera with the graphical prompt information, and displaying the superimposed image on the display screen of the electronic rearview mirror.

[0109] As one possible implementation of this application, the fault diagnosis unit 71 includes: The first diagnostic module is used to monitor the interruption signal of the video frame of the camera, or to monitor the image frame sequence number count of the camera; when the interruption signal of the video frame is detected within a continuous preset time period, or when the image frame sequence number count stops updating within a continuous preset time period, the first type of fault of the camera is diagnosed.

[0110] As one possible implementation of this application, the fault diagnosis unit 71 further includes: The second diagnostic module is used to calculate the sharpness of the image output by the camera; when the sharpness is lower than a preset sharpness threshold, the camera is diagnosed as having a second type of fault.

[0111] As one possible implementation of this application, the second diagnostic module is further configured to: The image output by the camera is divided into several image regions, and the average grayscale value of each image region is calculated. When the average grayscale value of an image region differs from the average grayscale value of other image regions in the same frame by more than a preset difference threshold, and this difference persists for several consecutive frames, the camera is diagnosed as having a second type of fault.

[0112] As one possible implementation of this application, the first fault processing unit 72 includes: The perception data acquisition module is used to acquire environmental perception data from radar and / or V2X vehicle-to-everything (V2X) modules; The contour determination module is used to determine the contour of the vehicle based on its inherent dimensional parameters; and to determine the contour of surrounding targets based on the environmental perception data. The environment view generation module is used to generate a graphical environment view of the side and rear of the vehicle based on the outline of the vehicle and the outline of the surrounding targets.

[0113] As one possible implementation of this application, when the environmental perception data comes from radar, the environmental perception data includes target data output by the radar, and the contour determination module is specifically used for: Analyze the target data to obtain the original coordinates of surrounding targets in the radar coordinate system; The original coordinates are transformed to the vehicle coordinate system using a pre-calibrated coordinate transformation matrix to obtain the coordinates of the surrounding targets in the vehicle coordinate system. The outline of the surrounding targets is determined based on their coordinates in the vehicle coordinate system.

[0114] As one possible implementation of this application, when the environmental perception data comes from a V2X vehicle-to-everything (V2X) module, the environmental perception data includes the state information of surrounding targets, and the contour determination module is specifically used for: Based on the latitude and longitude coordinates of the surrounding targets and the real-time latitude and longitude coordinates of the vehicle, the coordinates of the surrounding targets in the vehicle coordinate system are determined through geodetic coordinate transformation; The outline of the surrounding targets is determined based on the coordinates of the surrounding targets in the vehicle coordinate system, the heading angle, and the vehicle dimensions.

[0115] As one possible implementation of this application, the step of overlaying the image captured by the camera with the graphical prompt information includes: The perspective projection model, established by pre-calibrating the internal and external parameters of the camera, maps the position of the target perceived by the non-image sensor in the vehicle coordinate system to the corresponding pixel coordinates in the image captured by the camera. At the position corresponding to the pixel coordinates, the graphical prompt information is overlaid on the image as a graphical layer.

[0116] As one possible embodiment of this application, the electronic rearview mirror image processing device further includes: The target information acquisition unit is used to acquire the relative distance and relative speed between the surrounding targets and the vehicle based on the environmental perception data from the non-image sensor. A risk level determination unit is used to determine the risk level of the surrounding targets based on the relative distance and the relative speed; The prompt attribute configuration unit is used to configure different prompt attributes for the graphical environment view or the graphical prompt information according to the risk level.

[0117] As can be seen from the above, in this embodiment, by performing real-time fault diagnosis on the camera, the fault type is accurately identified. When a first type of fault is diagnosed, a graphical environmental view of the vehicle's side and rear is generated based on environmental perception data from non-image sensors. This view completely replaces the failed camera image on the display screen, providing the driver with an uninterrupted, graphical, direct view and avoiding safety hazards caused by complete loss of vision. When a second type of fault is diagnosed, graphical prompts are generated based on environmental perception data from non-image sensors. The image captured by the camera is superimposed on this graphical prompt, and the superimposed image is displayed on the display screen. By superimposing graphical prompts on the camera image, the distortion and incompleteness of the fault image can be accurately compensated for, effectively enhancing the driver's ability to identify key environmental information in the image. This application implements differentiated display strategies for different fault categories, ensuring that the driver can consistently and reliably obtain key environmental information from the electronic rearview mirror display screen when the camera malfunctions, thereby effectively reducing driving risks.

[0118] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0119] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 6 The steps of any electronic rearview mirror image processing method are represented.

[0120] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 6 The steps of any electronic rearview mirror image processing method are represented.

[0121] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the following: Figures 1 to 6 The steps of any electronic rearview mirror image processing method are represented.

[0122] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 8As shown, the electronic device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the various embodiments of the electronic rearview mirror image processing method described above, for example... Figure 1 Steps S101 to S103 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of units 71 to 73 are shown.

[0123] For example, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program 82 in the electronic device 8.

[0124] The electronic device 8 may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 8 may also include input / output devices, network access devices, buses, etc.

[0125] The processor 80 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0126] The memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or memory. The memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 81 can include both internal and external storage units of the electronic device 8. The memory 81 is used to store the computer program and other programs and data required by the electronic device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0127] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0131] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method for an electronic rearview mirror, characterized in that, include: Real-time fault diagnosis of the camera in the electronic rearview mirror; In response to the diagnosis of a first type of fault, a first display strategy is executed, the first display strategy including: generating a graphical environmental view of the side and rear of the vehicle based on environmental perception data from a non-image sensor, and displaying the graphical environmental view on the display screen of the electronic rearview mirror; In response to the diagnosis of a second type of fault, a second display strategy is executed, which includes: generating a graphical prompt based on environmental perception data from a non-image sensor, overlaying the image captured by the camera with the graphical prompt, and displaying the overlaid image on the display screen of the electronic rearview mirror.

2. The method according to claim 1, characterized in that, The real-time fault diagnosis of the camera in the electronic rearview mirror includes: Monitor the interruption signal of the video frame of the camera, or monitor the sequence number count of the image frame of the camera; When an interruption signal of the video frame is detected within a consecutive preset time period, or when the image frame sequence number count stops updating within a consecutive preset time period, the camera is diagnosed as having a first type of fault.

3. The method according to claim 1, characterized in that, The real-time fault diagnosis of the camera in the electronic rearview mirror includes: Calculate the sharpness of the image output by the camera; When the resolution is lower than a preset resolution threshold, a second type of fault is diagnosed in the camera.

4. The method according to claim 1, characterized in that, The real-time fault diagnosis of the camera in the electronic rearview mirror includes: The image output by the camera is divided into several image regions, and the average grayscale value of each image region is calculated. When the average grayscale value of an image region differs from the average grayscale value of other image regions in the same frame by more than a preset difference threshold, and this difference persists for several consecutive frames, the camera is diagnosed as having a second type of fault.

5. The method according to claim 1, characterized in that, The generation of a graphical environmental view of the vehicle's side and rear based on environmental perception data from non-image sensors includes: Acquire environmental perception data from radar and / or V2X vehicle-to-everything (V2X) modules; The outline of this vehicle is determined based on its inherent dimensional parameters; Based on the environmental perception data, the outlines of surrounding targets are determined; Based on the outline of the vehicle and the outlines of the surrounding targets, a graphical environment view of the side and rear of the vehicle is generated.

6. The method according to claim 5, characterized in that, When the environmental perception data comes from radar, the environmental perception data includes target data output by the radar. Determining the contours of surrounding targets based on the environmental perception data includes: Analyze the target data to obtain the original coordinates of surrounding targets in the radar coordinate system; The original coordinates are transformed to the vehicle coordinate system using a pre-calibrated coordinate transformation matrix to obtain the coordinates of the surrounding targets in the vehicle coordinate system. The outline of the surrounding targets is determined based on their coordinates in the vehicle coordinate system.

7. The method according to claim 5, characterized in that, When the environmental perception data comes from the V2X vehicle-to-everything (V2X) module, the environmental perception data includes the status information of surrounding targets, and the status information includes the latitude and longitude coordinates, heading angle, and vehicle size of the surrounding targets. Determining the contours of surrounding targets based on the environmental perception data includes: Based on the latitude and longitude coordinates of the surrounding targets and the real-time latitude and longitude coordinates of the vehicle, the coordinates of the surrounding targets in the vehicle coordinate system are determined through geodetic coordinate transformation; The outline of the surrounding targets is determined based on the coordinates of the surrounding targets in the vehicle coordinate system, the heading angle, and the vehicle dimensions.

8. The method according to any one of claims 1 to 7, characterized in that, The step of overlaying the image captured by the camera with the graphical prompt information includes: The perspective projection model, established by pre-calibrating the internal and external parameters of the camera, maps the position of the target perceived by the non-image sensor in the vehicle coordinate system to the corresponding pixel coordinates in the image captured by the camera. At the position corresponding to the pixel coordinates, the graphical prompt information is overlaid on the image as a graphical layer.

9. An electronic rearview mirror image processing device, characterized in that, include: The fault diagnosis unit is used to perform real-time fault diagnosis on the camera of the electronic rearview mirror. The first fault processing unit is configured to execute a first display strategy in response to the diagnosis of a first type of fault. The first display strategy includes: generating a graphical environmental view of the side and rear of the vehicle based on environmental perception data from a non-image sensor, and displaying the graphical environmental view on the display screen of the electronic rearview mirror. The second fault processing unit is used to execute a second display strategy in response to the diagnosis of a second type of fault. The second display strategy includes: generating graphical prompt information based on environmental perception data from a non-image sensor, overlaying the image captured by the camera with the graphical prompt information, and displaying the overlaid image on the display screen of the electronic rearview mirror.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the electronic rearview mirror image processing method as described in any one of claims 1 to 7.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the electronic rearview mirror image processing method as described in any one of claims 1 to 7.

Images