Blind spot alert method for rearview mirror display screen, and rearview mirror display screen

Abstract

A blind spot alert method for a rearview mirror display screen, and a rearview mirror display screen, relating to the field of monitoring and alert display. The blind spot alert method for a rearview mirror display screen comprises: receiving an external blind spot image generated by a vehicle monitoring apparatus, and performing pre-processing on the external blind spot image; inputting the processed external blind spot image into a pedestrian detection model, the pedestrian detection model performing inference and judgment on the external blind spot image and outputting an image marked with a pedestrian marking box; a rearview mirror display screen displaying, to a driver, the image marked with the pedestrian marking box. Training steps of the pedestrian detection model comprise: acquiring a safe distance of a vehicle and a set of vehicle surrounding images containing marking data; generating an initial marking box on the basis of the marking data; and inputting an image having the initial marking box into a YOLO network for training until the YOLO network converges. The method can cause a driver to efficiently and conveniently judge the distance of a pedestrian from the vehicle by means of the rearview mirror display screen, thereby improving driving safety and comfort.

Description

A rearview mirror display screen blind spot alarm method and rearview mirror display screen Technical Field

[0001] The embodiments of this specification relate to the field of monitoring display alarms, and in particular to a rearview mirror display screen blind spot alarm method and a rearview mirror display screen. Background Art

[0002] In recent years, with the rapid development of smart cars and deep learning technology, deep learning-based object recognition algorithms have been used to identify objects around the car. These algorithms can accurately identify various objects outside the car, assisting the driver in making decisions and significantly improving driving safety and efficiency. Existing electronic rearview mirror displays monitor the exterior of the car in real time using a camera mounted on the outside of the car. This allows the driver to observe the surroundings of the car to the greatest extent possible and make informed decisions. However, existing technologies simply display and frame objects on the electronic rearview mirror display, making it difficult for the driver to intuitively determine the distance between the objects shown on the electronic rearview mirror display and the vehicle itself, leading to inaccurate collision risk assessments. Summary of the Invention

[0003] The embodiments of this specification provide a rearview mirror display screen blind spot alarm method and a rearview mirror display screen, aiming to solve one or more of the above-mentioned problems and other potential problems.

[0004] To achieve the above objectives, the following technical solutions are provided:

[0005] According to a first aspect of this specification, a rearview mirror display blind spot alarm method is provided, comprising:

[0006] The rearview mirror display screen receives an image of the blind spot outside the vehicle generated by the vehicle monitoring device, pre-processes the image, and inputs the processed image into a pedestrian detection model. The pedestrian detection model infers and judges the image of the blind spot outside the vehicle and outputs an image with a pedestrian annotation frame drawn on it. The rearview mirror display screen displays the image with the pedestrian annotation frame drawn on it to the driver; wherein the pedestrian annotation frame selects pedestrians who enter a safe distance from the vehicle;

[0007] The training steps of the pedestrian detection model include: obtaining an image set containing labeled data of a safe distance from a car and the surroundings of the car; generating an initial labeled box on the pictures in the image set based on the labeled data; inputting the pictures with the initial labeled box into the YOLO network for training until the YOLO network converges to obtain the trained pedestrian detection model; wherein the labeled data includes distance labels and category labels, and the initial labeled box selects pedestrians in the pictures in the image set that are within the safe distance from the car.

[0008] The rearview mirror display screen blind spot alarm method provided in the embodiments of this specification combines the rearview mirror display screen with the YOLO series neural network algorithm, identifies and predicts the position of pedestrians based on the global information of the image, and adds pedestrian distance parameters to accurately frame pedestrians who enter the safe distance of the car. Pedestrians who enter the safe distance of the car are intuitively marked on the rearview mirror display screen. The inference speed is fast and the generalization ability is strong, so that the driver can make timely judgments to reduce the incidence of car accidents.

[0009] In some embodiments, the image with the initial annotation box is input into the YOLO network for training until the YOLO network converges to obtain the trained pedestrian detection model, including:

[0010] The YOLO network extracts and fuses features from the image with the initial annotation frame, and uses a classification and regression network to obtain the position of pedestrians in the image with the initial annotation frame;

[0011] Based on the distance annotation, it is determined whether the pedestrian in the picture with the initial annotation box is within the safe distance of the car. If the pedestrian is within the safe distance of the car, the pedestrian at the position is selected using the predicted annotation box to obtain a picture with the predicted annotation box;

[0012] The predicted annotation box is compared with the initial annotation box to obtain a loss value, and the parameters of the YOLO network are adjusted based on the loss value until the loss value is reduced to a preset value and the YOLO network converges.

[0013] In some embodiments, when the pedestrian detection model makes inferences and judgments on the blind spot image outside the vehicle, continuous inference is performed in combination with the previous frame image of the blind spot image outside the vehicle in time.

[0014] In some embodiments, performing continuous reasoning on the images before and after the blind spot image outside the vehicle in time includes:

[0015] Obtaining an inference result of a pedestrian annotation box of a previous frame image of the blind spot image outside the vehicle;

[0016] Comparing the blind spot image outside the vehicle with the previous frame image to obtain difference features, and performing inference judgment on the difference features;

[0017] Based on the inference and judgment result of the difference feature, the pedestrian labeling frame of the previous frame image of the external blind spot image is adjusted to obtain the pedestrian labeling frame of the external blind spot image.

[0018] In some embodiments, pre-processing the blind spot image outside the vehicle includes:

[0019] Environmental recognition step: Based on the recognized environmental scene, different noise reduction processing steps are performed; wherein the noise reduction processing steps include image enhancement and defogging steps for foggy scenes and image exposure adaptation steps for bright light scenes;

[0020] Foreground extraction step: removing background features that do not need to be analyzed and processed in the blind spot image outside the vehicle, and extracting foreground features.

[0021] In some embodiments, when the rearview mirror display screen displays an image with a pedestrian marking frame drawn on it to the driver, it obtains the brightness information of the ambient light inside the vehicle and adjusts its own display brightness based on the brightness information of the ambient light inside the vehicle.

[0022] In some embodiments, when the rearview mirror display screen displays an image with a pedestrian marking frame drawn therein to the driver, the driver's facial orientation information is also obtained, and the display angle of the rearview mirror display screen is adjusted based on the facial orientation information.

[0023] According to a second aspect of the present specification, a rearview mirror display screen is provided, comprising a pre-processing module, a control module and a display alarm module; the pre-processing module is used to pre-process an image of a blind spot outside the vehicle; the control module analyzes and identifies the processed image of the blind spot outside the vehicle, and outputs an image with a pedestrian marking frame drawn therein; the display alarm module is used to display the image with the pedestrian marking frame drawn therein to the driver, and issue a warning to the driver; wherein the control module deploys the pedestrian detection model in the above-mentioned method for blind spot alarm of a rearview mirror display screen.

[0024] In some embodiments, the pedestrian detection model is deployed in the control module using an NCNN deployment framework.

[0025] In some embodiments, the display alarm module includes a base, a display screen and a tracking sensing device, and the base and the display screen are movably connected; the tracking sensing device senses the driver's facial orientation information and the ambient light brightness information in the vehicle, and sends the driver's facial orientation information and the ambient light brightness information in the vehicle to the control module; the control module controls the display screen to move along the base based on the driver's facial orientation information, so that the display screen faces the driver's face; the control module controls the brightness of the display screen based on the ambient light brightness information in the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the embodiments of the present specification will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present specification are shown by way of example and not limitation.

[0027] FIG1 shows a flow chart of a rearview mirror display screen blind spot alarm method according to an embodiment of the present specification;

[0028] FIG2 shows a flow chart of the training steps of the pedestrian detection model in FIG1 ;

[0029] FIG3 shows a schematic diagram of the installation angle of the front camera according to an embodiment of this specification;

[0030] FIG4 is a schematic diagram showing the installation angle of the lower camera according to an embodiment of the present specification.

[0031] 1-Front camera, 2-Lower camera.

[0032] In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0034] The term "including" and its variations used in this document indicate open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one other embodiment". Terms such as "upper", "lower", "front", and "rear" indicating placement or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the principles of this specification, and do not indicate or imply that the referred elements must have a specific orientation, be constructed or operate in a specific orientation, and therefore should not be understood as limiting this specification.

[0035] The following describes in detail a rearview mirror display screen blind spot alarm method and a rearview mirror display screen according to an embodiment of the present invention with reference to the accompanying drawings. FIG1 shows a flow chart of a rearview mirror display screen blind spot alarm method according to an embodiment of the present invention. A rearview mirror display screen blind spot alarm method according to an embodiment of the present invention includes:

[0036] S1. The rearview mirror display screen receives an image of the blind spot outside the vehicle generated by a vehicle monitoring device and performs pre-processing on the image;

[0037] S3, inputting the processed blind spot image outside the vehicle into a pedestrian detection model, wherein the pedestrian detection model performs inference and judgment on the blind spot image outside the vehicle and outputs an image with a pedestrian annotation box drawn thereon;

[0038] S5. The rearview mirror display screen displays an image with a pedestrian marking frame drawn thereon to the driver.

[0039] In step S1, the image of the blind spot outside the vehicle is pre-processed, including:

[0040] S100, environment recognition step: performing different noise reduction processing steps based on the recognized environment scene;

[0041] S102, foreground extraction step: removing background features that do not need to be analyzed and processed in the blind spot image outside the vehicle, and extracting foreground features.

[0042] During car driving, the external environment is relatively complex. Various factors such as haze, light, and dust will affect the clarity of images taken around the car. Corresponding processing methods are needed to improve image quality and then perform inference detection. Therefore, in step S100, the environment in which the blind spot image outside the vehicle is captured is identified, and different noise reduction processing steps are performed for different shooting environments. For example, in foggy conditions, the blind spot image outside the vehicle is processed through an image enhancement and defogging step. When the shooting light is too strong, resulting in overexposure of the image, the blind spot image outside the vehicle is processed through an image exposure adaptation step. The foggy environment can be identified according to an existing image fog detection algorithm. For example, the grayscale image of the blind spot image outside the vehicle is detected. The grayscale histogram of a fog-free image is relatively evenly distributed, while the grayscale histogram of a foggy image is unevenly distributed due to different brightness and darkness distributions of the image. Similarly, whether the image is overexposed can also be identified by histogram detection and other methods. There is a peak on the right side of the histogram of the overexposed image, but the rightmost part is flatter, and the dark area of ​​the picture lacks details. The histogram pixels of the underexposed image are concentrated in the shadow area on the left, and the brightness is relatively dim. This step performs adaptive pre-processing on the blind spot image outside the vehicle, improves the quality of the blind spot image outside the vehicle, provides good conditions for subsequent reasoning analysis, reduces reasoning errors, and makes the image clearer and more vivid for the driver to observe.

[0043] In step 102, the foreground of the blind spot image outside the vehicle is extracted by background subtraction, edge detection, etc., the moving object is separated from the static background, the moving objects around the car are located, and the interference of background features is reduced.

[0044] In step S3, when the pedestrian detection model performs reasoning and judgment on the blind spot image outside the vehicle, continuous reasoning is performed in combination with the previous frame image of the blind spot image outside the vehicle in time, specifically including:

[0045] S300, obtaining the pedestrian labeling frame inference result of the previous frame image of the blind spot image outside the vehicle;

[0046] S302, comparing the blind spot image outside the vehicle with the previous frame image to obtain difference features, and performing inference and judgment on the difference features;

[0047] S304 : Based on the inference and judgment result of the difference feature, adjust the pedestrian labeling frame of the previous frame image of the external blind spot image to obtain the pedestrian labeling frame of the external blind spot image.

[0048] In reality, pedestrians outside cars are constantly moving. The blind spot image outside the vehicle is captured from the video captured by the camera. The image to be detected and the frame immediately preceding it share most of the same features. Therefore, when inferring the model, the previous frame of the image to be detected can be referenced for inference, with only the differences analyzed, reducing computational effort and improving recognition accuracy. In another implementation, inference can be performed separately on the image to be detected and the frame immediately preceding it, and the inference results can then be cross-checked. If the inference results differ significantly, an abnormality warning signal can be issued promptly.

[0049] In step S304, the pedestrian annotation frame of the previous frame image of the external blind spot image is adjusted. Specifically, the pedestrian annotation frame of the same part of the external blind spot image and its previous frame image is retained, and an annotation frame inferred based on the difference features is added to the pedestrian annotation frame of the same part to form the final pedestrian annotation frame of the external blind spot image.

[0050] The training steps of the pedestrian detection model in step S3 include:

[0051] S31, obtaining a safe distance from the car and an image set containing labeled data around the car;

[0052] S33, generating an initial annotation frame on the picture in the image set based on the annotation data;

[0053] S35, inputting the picture with the initial annotation box into the YOLO network for training until the YOLO network converges, thereby obtaining the trained pedestrian detection model;

[0054] In step 31, the images in the image set are preprocessed, including flipping, rotation, color dithering, grayscale, overlay, blocking, brightness, contrast and noise processing steps, so as to increase the complexity of the pedestrian detection model training images, simulate external interference on the images, enhance the generalization ability of the pedestrian detection model and the adaptability to complex real scenes, so as to achieve a more accurate model inference effect.

[0055] When the initial annotation frame is generated in step S33, the initial annotation frame may be drawn manually, or may be drawn based on the annotation data using artificial intelligence or a computer algorithm.

[0056] The backbone of the pedestrian detection model utilizes ShuffleNetV2, which features a small number of parameters, minimal resource usage, excellent single-core performance, and lower power consumption, enabling real-time inference performance on less powerful chips. Based on extensive experiments, the model's multi-scale configuration and network channel number have been optimized to address the problem of identifying objects of varying scales within the visible range, enhancing the cost-effectiveness of real-time inference at the edge.

[0057] As shown in FIG2 , step S35 specifically includes:

[0058] S351, YOLO network performs feature extraction and fusion on the picture with the initial annotation frame, and uses classification and regression network to obtain the position of pedestrians in the picture with the initial annotation frame;

[0059] S353: Determine, based on the distance annotation, whether the pedestrian in the image with the initial annotation frame is within a safe distance for a car; if so, select the pedestrian at that location using a predicted annotation frame to obtain a picture with the predicted annotation frame;

[0060] S355: Compare the predicted annotation box with the initial annotation box to obtain a loss value, and adjust the parameters of the YOLO network based on the loss value until the loss value is reduced to a preset value and the YOLO network converges.

[0061] In step S353, the distance to the target pedestrian in the image is obtained through classification and regression networks, and then the distance is determined to be within a safe distance for the car. When the predicted annotation box is used for selection, the optimization configuration is performed based on the KMEANS clustering algorithm.

[0062] In step S355, the loss value may be the signal-to-noise ratio between the predicted annotation box and the initial annotation box. The smaller the difference between the predicted annotation box and the initial annotation box, the lower the loss value, and the more accurate the inference result of the pedestrian detection model.

[0063] In step S5, when the rearview mirror display screen displays the image with the pedestrian marker frame to the driver, it obtains information about the ambient light brightness within the vehicle and adjusts its display brightness based on this information. It also obtains information about the driver's facial orientation and adjusts the display angle of the rearview mirror display screen based on this information. The rearview mirror display screen is movable, adjusting its position based on the driver's facial orientation and the ambient light brightness within the vehicle. This eliminates the driver's need to turn their head significantly to view the image, saving time and effort and improving driving safety. The rearview mirror display screen's brightness is automatically adjusted based on the ambient light brightness within the vehicle, allowing the driver to receive the image on the rearview mirror display screen more clearly, resulting in a more vivid and comfortable viewing experience.

[0064] An embodiment of the present specification also provides a rearview mirror display screen, comprising a pre-processing module, a control module, and a display alarm module; the pre-processing module is used to pre-process the blind spot image outside the vehicle; the control module analyzes and identifies the processed blind spot image outside the vehicle, and outputs an image with a pedestrian marking box drawn on it; the display alarm module is used to display the image with the pedestrian marking box drawn on it to the driver, and issue a warning to the driver; wherein the control module deploys the pedestrian detection model in the above-mentioned rearview mirror display screen blind spot alarm method.

[0065] The blind spot image outside the vehicle can be captured by the cameras installed on both sides of the vehicle. The cameras include a front camera 1 and a lower camera 2. As shown in Figures 3 and 4, the shooting angle ranges of the front camera 1 and the lower camera 2 are respectively shown. The lens of the front camera 1 is horizontally facing the rear of the vehicle and deflected about 30° toward the ground. The lens of the lower camera 2 is horizontally facing the ground and deflected about 15° toward the rear of the vehicle. The front camera 1 and the lower camera 2 are respectively installed on pillars of the vehicle 2.5m and 2m above the ground. When the display alarm module displays an image with a pedestrian marking box to the driver, the display screen is divided into a rear view area and a lower view area, respectively displaying the images captured by the front camera 1 and the lower camera 2. The control module sends information to the display alarm module for processing via a queue such as http or kafka.

[0066] The pedestrian detection model is deployed in the control module using the NCNN deployment framework. The NCNN deployment framework is a lightweight, high-performance deep learning framework that uses optimized computational graphs and memory management strategies to minimize memory usage and computational overhead, providing efficient inference performance and low memory usage. Furthermore, the NCNN deployment framework supports multiple operating systems and hardware platforms, running on different processor architectures, and has a wider range of applicability.

[0067] The display alarm module includes a base, a display screen, and a tracking sensor device, wherein the base and the display screen are movably connected. The tracking sensor device senses the driver's facial orientation information and the ambient light brightness information within the vehicle, and transmits the driver's facial orientation information and the ambient light brightness information within the vehicle to the control module. The control module controls the display screen to move along the base based on the driver's facial orientation information, so that the display screen faces the driver's face. The control module controls the brightness of the display screen based on the ambient light brightness information within the vehicle. The tracking sensor device includes hardware such as a photosensitive device and a camera. The photosensitive device detects the ambient light brightness within the vehicle and transmits a brightness signal to the control module. The control module adjusts the display screen brightness based on the brightness signal. The higher the ambient light brightness within the vehicle, the higher the display screen brightness. The camera captures the driver's head image in real time and transmits the driver's head image to the control module. The control module performs facial recognition on the driver's head image, locates the pupil position, and then controls the display screen to rotate toward the driver's pupil position.

[0068] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this specification. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0069] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0070] While the embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A rearview mirror display blind spot alarm method, characterized in that: include: The rearview mirror display screen receives an image of the blind spot outside the vehicle generated by the vehicle monitoring device, pre-processes the image, and inputs the processed image into a pedestrian detection model. The pedestrian detection model infers and judges the image of the blind spot outside the vehicle and outputs an image with a pedestrian annotation frame drawn on it. The rearview mirror display screen displays the image with the pedestrian annotation frame drawn on it to the driver; wherein the pedestrian annotation frame selects pedestrians who enter a safe distance from the vehicle; The training steps of the pedestrian detection model include: obtaining an image set containing labeled data of a safe distance from a car and the surroundings of the car; generating an initial labeled box on the pictures in the image set based on the labeled data; inputting the pictures with the initial labeled box into the YOLO network for training until the YOLO network converges to obtain the trained pedestrian detection model; wherein the labeled data includes distance labels and category labels, and the initial labeled box selects pedestrians in the pictures in the image set that are within the safe distance from the car.

2. The method according to claim 1, characterized in that The image with the initial annotation box is input into the YOLO network for training until the YOLO network converges to obtain the trained pedestrian detection model, including: The YOLO network extracts and fuses features from the image with the initial annotation frame, and uses a classification and regression network to obtain the position of pedestrians in the image with the initial annotation frame; Based on the distance annotation, it is determined whether the pedestrian in the picture with the initial annotation box is within the safe distance of the car. If the pedestrian is within the safe distance of the car, the pedestrian at the position is selected using the predicted annotation box to obtain a picture with the predicted annotation box; The predicted annotation box is compared with the initial annotation box to obtain a loss value, and the parameters of the YOLO network are adjusted based on the loss value until the loss value is reduced to a preset value and the YOLO network converges.

3. The method according to claim 1, characterized in that When the pedestrian detection model makes inferences and judgments on the blind spot image outside the vehicle, continuous inference is performed in combination with the previous frame image of the blind spot image outside the vehicle in time.

4. The method according to claim 3, characterized in that Performing continuous reasoning based on the temporally preceding and following images of the blind spot image outside the vehicle includes: Obtaining an inference result of a pedestrian annotation box of a previous frame image of the blind spot image outside the vehicle; Comparing the blind spot image outside the vehicle with the previous frame image to obtain difference features, and performing inference judgment on the difference features; Based on the inference and judgment result of the difference feature, the pedestrian labeling frame of the previous frame image of the external blind spot image is adjusted to obtain the pedestrian labeling frame of the external blind spot image.

5. The method according to claim 1, wherein Pre-processing the blind spot image outside the vehicle includes: Environmental recognition step: Based on the recognized environmental scene, different noise reduction processing steps are performed; wherein the noise reduction processing steps include image enhancement and defogging steps for foggy scenes and image exposure adaptation steps for bright light scenes; Foreground extraction step: removing background features that do not need to be analyzed and processed in the blind spot image outside the vehicle, and extracting foreground features.

6. The method according to claim 1, characterized in that When the rearview mirror display screen displays the image with the pedestrian marking frame drawn on it to the driver, it obtains the brightness information of the ambient light inside the vehicle and adjusts its own display brightness based on the brightness information of the ambient light inside the vehicle.

7. The method according to claim 1, characterized in that When the rearview mirror display screen displays the image with the pedestrian marking frame drawn on it to the driver, it also obtains the driver's facial orientation information and adjusts the display angle of the rearview mirror display screen based on the facial orientation information.

8. Rearview mirror display screen, characterized in that, It includes a pre-processing module, a control module and a display alarm module; the pre-processing module is used to pre-process the blind spot image outside the vehicle; the control module analyzes and identifies the processed blind spot image outside the vehicle, and outputs an image with a pedestrian marking box drawn; the display alarm module is used to display the image with the pedestrian marking box drawn to the driver and issue a warning to the driver; wherein, the pedestrian detection model according to claim 1 is deployed in the control module.

9. The rearview mirror display screen according to claim 8, characterized in that: The pedestrian detection model is deployed in the control module using the NCNN deployment framework.

10. The rearview mirror display screen according to claim 8, characterized in that: The display alarm module includes a base, a display screen, and a tracking sensor device, wherein the base and the display screen are movably connected; the tracking sensor device senses the driver's facial orientation information and the ambient light brightness information in the vehicle, and sends the driver's facial orientation information and the ambient light brightness information in the vehicle to the control module; The control module controls the display screen to move along the base based on the driver's face orientation information so that the display screen faces the driver's face; and the control module controls the brightness of the display screen based on the in-vehicle ambient light brightness information.

Images