Vehicle blind area monitoring method, device and equipment and storage medium

Abstract

The invention discloses a vehicle blind area monitoring method, device and equipment and a storage medium, and relates to the technical field of vehicles, and the vehicle blind area monitoring method comprises the steps: carrying out the preprocessing of a camera monitoring image, and obtaining a target splicing image; obtaining a target tracking result according to the target spliced image, a target detection model and a preset target tracking algorithm; inputting the target spliced image to a target ranging model to obtain a target positioning result; and determining a blind area monitoring result according to the target tracking result and the target positioning result. By calibrating the internal and external parameters of the camera, acquiring and preprocessing the camera monitoring image, tracking the target and measuring the distance, and finally obtaining the blind area monitoring result, the blind area monitoring efficiency is improved while the computing power consumption of the vehicle end controller is reduced.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle blind spot monitoring method, device, equipment and storage medium. Background Art

[0002] Blind spot monitoring system (BSD), also known as lane change assistance system, mainly detects the blind spot of the rearview mirror of the car through the detector. When there is a vehicle in the blind spot of the field of vision, it will issue warnings such as light, sound, vibration, etc. for the behavior of preparing to change lanes, overtaking, etc., to remind the driver, so as to avoid traffic accidents and improve driving safety. Among the number of deaths caused by traffic accidents in my country each year, traffic accidents caused by lane changes account for about 70%, the highest proportion, so the importance of research and application of blind spot monitoring system is prominent. Blind spot monitoring system can be used in the intelligent upgrade of ordinary cars, and can also be used in the field of unmanned vehicles. In addition, commercial vehicles, especially large buses and large trucks with trailers, have large blind spots in the driver's field of vision due to their long body and large volume, especially the blind spots near the front of commercial vehicles and right turns. In the large number of traffic accidents caused in recent years, heavy vehicles have caused the highest proportion of serious casualties. However, the existing blind spot monitoring systems mostly use radar or fusion perception with visual sensors, which requires joint calibration and feature extraction fusion of the two, which is not only time-consuming but also has high hardware costs. In addition, another blind spot monitoring method based on left and right rear-view cameras can only provide early warning for the blind spot behind the vehicle. In general, the blind spot monitoring system is of great significance to driving safety and has broad market space in the future. Summary of the invention

[0003] The main purpose of this application is to provide a vehicle blind spot monitoring method, device, equipment and storage medium, aiming to solve the technical problem of how to improve monitoring efficiency while reducing the computing power consumption of the vehicle-side controller.

[0004] To achieve the above objectives, the present application proposes a vehicle blind spot monitoring method, the vehicle blind spot monitoring method comprising:

[0005] Preprocess the camera monitoring image to obtain the target stitching image;

[0006] Obtaining a target tracking result according to the target spliced ​​image, the target detection model and the preset target tracking algorithm;

[0007] Inputting the target stitching image into the target ranging model to obtain a target positioning result;

[0008] The blind spot monitoring result is determined according to the target tracking result and the target positioning result.

[0009] In one embodiment, the preprocessing of the camera monitoring image to obtain the target stitching image includes:

[0010] Decoding the camera monitoring image to obtain a decoded camera monitoring image;

[0011] Correcting the decoded camera monitoring image according to camera internal parameters and distortion parameters to obtain a corrected monitoring image;

[0012] The corrected monitoring images are stitched from a bird's-eye view according to the external parameter matrix to obtain a target stitched image.

[0013] In one embodiment, inputting the target stitched image into the target ranging model to obtain the target positioning result includes:

[0014] Inputting the target stitching image into the overhead distance measurement model in the target distance measurement model according to the overhead transformation scale information to obtain the overhead distance measurement information;

[0015] Inputting the target stitching image into a backsight ranging model in a target ranging model according to an external parameter matrix and a ground constraint relationship to obtain backsight ranging information;

[0016] A target positioning result is obtained according to the overhead distance measurement information and the rearward distance measurement information.

[0017] In one embodiment, before obtaining the target tracking result according to the target stitched image, the target detection model and the preset target tracking algorithm, the method further includes:

[0018] Annotate the blind area image data of a preset number of frames to obtain image sample data;

[0019] Training the initial detection model according to the image sample data to obtain a model training result;

[0020] A target detection model is obtained according to the model training results.

[0021] In one embodiment, obtaining a target tracking result according to the target stitched image, the target detection model and the preset target tracking algorithm includes:

[0022] Inputting the target stitching image into the target detection model to obtain the target detection result;

[0023] Target tracking is performed according to the target detection result and a preset target tracking algorithm to obtain a target tracking result.

[0024] In one embodiment, before preprocessing the camera monitoring image to obtain the target stitched image, the method further includes:

[0025] Calibrate the vehicle blind spot cameras at multiple angles according to the preset camera model to obtain the camera intrinsic parameters, distortion parameters and extrinsic parameter matrix;

[0026] Determine a camera coordinate transformation relationship according to the camera intrinsic parameters, the distortion parameters and the extrinsic parameter matrix;

[0027] The calibrated vehicle blind spot cameras at multiple angles are obtained according to the camera coordinate conversion relationship, and the camera monitoring images are collected by the calibrated vehicle blind spot cameras at multiple angles.

[0028] In one embodiment, before preprocessing the camera monitoring image to obtain the target stitched image, the method further includes:

[0029] Calibrate the vehicle blind spot cameras at multiple angles according to the preset camera model to obtain the camera intrinsic parameters, distortion parameters and extrinsic parameter matrix;

[0030] Determine a camera coordinate transformation relationship according to the camera intrinsic parameters, the distortion parameters and the extrinsic parameter matrix;

[0031] The calibrated vehicle blind spot cameras at multiple angles are obtained according to the camera coordinate conversion relationship, and the camera monitoring images are collected by the calibrated vehicle blind spot cameras at multiple angles.

[0032] In addition, to achieve the above purpose, the present application also proposes a vehicle blind spot monitoring device, the vehicle blind spot monitoring device comprising:

[0033] A processing module, used for preprocessing the camera monitoring image to obtain a target stitching image;

[0034] A tracking module, used to obtain a target tracking result according to the target spliced ​​image, the target detection model and a preset target tracking algorithm;

[0035] A positioning module, used for inputting the target stitching image into the target ranging model to obtain a target positioning result;

[0036] The monitoring module is used to determine the blind spot monitoring result according to the target tracking result and the target positioning result.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle blind spot monitoring device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the vehicle blind spot monitoring method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle blind spot monitoring method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the vehicle blind spot monitoring method described above are implemented.

[0040] This application obtains a target stitching image by preprocessing the camera monitoring image; obtains the target tracking result based on the target stitching image, the target detection model and the preset target tracking algorithm; inputs the target stitching image into the target ranging model to obtain the target positioning result; and determines the blind spot monitoring result based on the target tracking result and the target positioning result. By calibrating the internal and external parameters of the camera, obtaining the camera monitoring image and preprocessing it, and then tracking and ranging the target, and finally obtaining the blind spot monitoring result, it is achieved that the efficiency of blind spot monitoring is improved while reducing the computing power consumption of the vehicle-side controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flow chart of the first embodiment of the vehicle blind spot monitoring method of the present application;

[0044] Figure 2 A schematic diagram of the camera installation position and its field of view provided in Example 1 of the vehicle blind spot monitoring method of the present application;

[0045] Figure 3 A schematic diagram of the blind spot monitoring range provided in Example 1 of the vehicle blind spot monitoring method of the present application;

[0046] Figure 4 A flow chart of the second embodiment of the vehicle blind spot monitoring method of the present application;

[0047] Figure 5 A schematic diagram of a brief process of a vehicle blind spot monitoring method provided in Example 2 of the present application;

[0048] Figure 6 This is a schematic diagram of the module structure of a vehicle blind spot monitoring device according to an embodiment of the present application;

[0049] Figure 7Schematic diagram of the device structure of the hardware operating environment involved in the vehicle blind spot monitoring method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: preprocessing the camera monitoring image to obtain a target stitching image; obtaining a target tracking result based on the target stitching image, a target detection model and a preset target tracking algorithm; inputting the target stitching image into the target ranging model to obtain a target positioning result; and determining the blind spot monitoring result based on the target tracking result and the target positioning result.

[0054] Blind spot monitoring system (BSD), also known as lane change assistance system, mainly detects the blind spot of the rearview mirror of the car through the detector. When there is a vehicle in the blind spot, it will issue warnings such as light, sound, vibration, etc. for the behavior of preparing lane change, overtaking, etc., to remind the driver, so as to avoid traffic accidents and improve driving safety. Among the number of deaths in traffic accidents in my country each year, traffic accidents caused by lane change account for about 70%, the highest proportion, so the importance of research and application of blind spot monitoring system is prominent. Blind spot monitoring system can be used in the intelligent upgrade of ordinary cars, and can also be used in the field of unmanned vehicles. In addition, commercial vehicles, especially large buses and large trucks with trailers, have large blind spots in the driver's field of vision due to their long body and large volume, especially the blind spots near the front of commercial vehicles and right turns. In recent years, among the large number of traffic accidents caused, heavy vehicles have caused the highest proportion of serious casualties. Existing blind spot monitoring systems mostly use radar or fusion perception with visual sensors. The two need to be calibrated and feature extracted and fused, which is not only time-consuming but also has high hardware costs. In addition, another blind spot monitoring method based on left and right rear-view cameras can only provide early warning for the blind spot behind the vehicle. In general, the blind spot monitoring system is of great significance to driving safety and has broad market space in the future.

[0055] This application obtains a target stitching image by preprocessing the camera monitoring image; obtains the target tracking result based on the target stitching image, the target detection model and the preset target tracking algorithm; inputs the target stitching image into the target ranging model to obtain the target positioning result; and determines the blind spot monitoring result based on the target tracking result and the target positioning result. By calibrating the internal and external parameters of the camera, obtaining the camera monitoring image and preprocessing it, and then tracking and ranging the target, and finally obtaining the blind spot monitoring result, it is achieved that the efficiency of blind spot monitoring is improved while reducing the computing power consumption of the vehicle-side controller.

[0056] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a vehicle blind spot monitoring device capable of realizing the above functions, etc. The following takes the vehicle blind spot monitoring device as an example of the execution subject to illustrate this embodiment and the following embodiments.

[0057] Based on this, the present application embodiment provides a vehicle blind spot monitoring method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the vehicle blind spot monitoring method of the present application.

[0058] In this embodiment, the vehicle blind spot monitoring method includes steps S10 to S40:

[0059] Step S10, preprocessing the camera monitoring image to obtain a target stitching image;

[0060] It should be noted that the camera installation position and its field of view are shown in the figure below. Figure 2 As shown, this embodiment adopts the method of pure vision with five cameras, which are symmetrically installed in front of the commercial vehicle and at the positions of the rearview mirrors, two of which are at the left and right rearview mirrors, and the cameras at the five positions are respectively recorded as front ①, left front ②, right front ③, left and right ④, and right rear ⑤.

[0061] It should be noted that the monitoring range diagram of this embodiment is as follows: Figure 3 As shown, according to the "Road Vehicle Blind Spot Monitoring (BSD) System Performance Requirements and Test Methods" issued by the Ministry of Industry and Information Technology in 2020, the blind spot definition range is the L1 and R1 areas shown in the figure. When the vehicle turns right, the blind spot definition range is the L1+R1+R2 area, and when a vehicle or pedestrian appears in the blind spot, an alarm signal should be immediately issued to the driver of the vehicle to achieve the purpose of early warning and avoid traffic accidents. The blind spot monitoring range of this technology is L1+R1+R2+Front. In comparison, the solution in this embodiment has a larger detection range.

[0062] It can be understood that the camera monitoring image refers to the blind spot monitoring image collected by each camera, and the target stitching image refers to the stitching image obtained by stitching multiple images from a bird's-eye view.

[0063] In a specific implementation, the blind spot monitoring images collected by each camera are subjected to dedistortion and bird's-eye view stitching processing, thereby obtaining a stitched image obtained by stitching multiple images from a bird's-eye view, namely, a target stitched image.

[0064] In a feasible implementation manner, step S10 may include steps A11 to A13:

[0065] Step A11, calibrating the vehicle blind spot cameras at multiple angles according to a preset camera model to obtain camera intrinsic parameters, distortion parameters and extrinsic parameter matrix;

[0066] It can be understood that the preset camera model refers to a pre-set fisheye camera model, the intrinsic camera parameters refer to the parameters that describe the camera imaging characteristics, mainly including focal length, principal point coordinates, etc., the distortion parameters are used to describe the image distortion caused by the limitations of the physical lens, and the extrinsic parameter matrix refers to the position and direction of the camera relative to the world coordinate system.

[0067] In a specific implementation, the five cameras of the vehicle in this embodiment are calibrated according to a preset fisheye camera model, thereby obtaining parameters describing the camera imaging characteristics, distortion parameters, and the position and direction of the camera relative to the world coordinate system.

[0068] Step A12, determining a camera coordinate transformation relationship according to the camera intrinsic parameters, the distortion parameters and the extrinsic parameter matrix;

[0069] It can be understood that the camera coordinate transformation relationship refers to the transformation relationship between the five cameras and the vehicle body coordinate system.

[0070] In a specific implementation, the parameters describing the imaging characteristics of the five cameras, the distortion parameters, and the positions and directions of the cameras relative to the world coordinate system are analyzed to obtain the conversion relationship between the five cameras and the vehicle body coordinate system.

[0071] Step A13, obtaining calibrated vehicle blind spot cameras at multiple angles according to the camera coordinate transformation relationship, and collecting camera monitoring images through the calibrated vehicle blind spot cameras at multiple angles.

[0072] It can be understood that after obtaining the conversion relationship between the five cameras and the vehicle body coordinate system, it means that the parameter calibration of the five cameras is completed, and then the images are collected by the calibrated five blind spot monitoring cameras to obtain five cameras.

[0073] Step S20, obtaining a target tracking result according to the target stitching image, the target detection model and the preset target tracking algorithm;

[0074] It can be understood that the target detection model refers to the target detection depth model, the preset target tracking algorithm includes Kalman filtering and Hungarian matching algorithm, etc., and the target tracking result refers to the target tracking result in the vehicle blind spot.

[0075] In the specific implementation, multiple images are input into the target detection depth model for top-down stitching to obtain the stitched image, and the category label and target rectangular frame output by the model are obtained. Then, the category label and target rectangular frame output by the model are tracked through the target tracking algorithm such as Kalman filtering or Hungarian matching algorithm, and finally the target tracking result in the vehicle blind spot is obtained.

[0076] In a feasible implementation manner, step S20 may include steps A21 to A23:

[0077] Step A21, annotating the blind area image data of a preset number of frames to obtain image sample data;

[0078] It can be understood that the preset frame number refers to the pre-set number of image frames, the blind spot image data refers to the image data collected by five cameras, and the image sample data refers to the sample data used for model training.

[0079] In the specific implementation, a real car equipped with five cameras is driven on the road, and continuous frame image data is captured at the same time. Then, data annotation is performed on traffic targets such as vehicles, pedestrians and bicycles in the image. The annotated images are then used as deep learning samples, that is, sample data for model training is obtained.

[0080] Step A22, training the initial detection model according to the image sample data to obtain a model training result;

[0081] It can be understood that the initial detection model refers to the initial target depth detection model, and the model training result refers to the result of whether the model training is completed.

[0082] In a specific implementation, the initial target depth detection model is trained according to the sample data used for model training to obtain a detection model with better performance, and then obtain the result of whether the model training is completed.

[0083] Step A23, obtaining a target detection model according to the model training result.

[0084] It can be understood that when the model training result is a training completion result, it indicates that a detection model with good performance has been obtained, and then the target detection model is determined.

[0085] Step S30, inputting the target stitching image into the target distance measurement model to obtain a target positioning result;

[0086] It can be understood that the target ranging model includes a top-view ranging model and a rear-view ranging model, and the target positioning result refers to the result of whether the blind spot target positioning is completed.

[0087] In a specific implementation, the stitched images obtained by stitching multiple images from a bird's-eye view are respectively input into the bird's-eye view ranging model and the rear-view ranging model, thereby obtaining the bird's-eye view ranging results and the rear-view ranging results output by the model, and then the bird's-eye view ranging results and the rear-view ranging results are summarized to obtain the target positioning result.

[0088] Step S40, determining a blind spot monitoring result according to the target tracking result and the target positioning result.

[0089] It can be understood that the blind spot monitoring result refers to the monitoring result of whether there is a target in the blind spot of the vehicle.

[0090] In a specific implementation, the target tracking and target positioning results, as well as time information, are combined to calculate the speed of the target relative to the vehicle, thereby determining the monitoring result of whether there is a target in the vehicle's blind spot.

[0091] In a feasible implementation manner, step S40 may include steps A41 to A43:

[0092] Step A41, when the blind spot monitoring result is that a traffic target exists, obtaining the current vehicle state;

[0093] It can be understood that the current vehicle state includes the vehicle straight-moving state and the vehicle turning state.

[0094] In a specific implementation, when the blind spot monitoring result is that there is a traffic target result, it indicates that there is a moving traffic target in the vehicle blind spot, and then the current driving state of the vehicle is determined to determine the corresponding alarm mode.

[0095] Step A42, when the current vehicle state is a vehicle straight-moving state, generating a first blind spot warning signal;

[0096] It can be understood that the first blind spot warning signal refers to the blind spot warning signal corresponding to the vehicle's straight-moving state.

[0097] In a specific implementation, when the current vehicle state is a straight-moving state, it indicates that the danger level of the traffic target traveling in the current blind spot is low, and a low-level alarm method is adopted, that is, a first blind spot alarm signal is generated, for example, only the instrument icon displays a warning.

[0098] Step A43, when the current vehicle state is a vehicle turning state, a second blind spot warning signal is generated, and the alarm level of the second blind spot warning signal is greater than the alarm level of the first blind spot warning signal.

[0099] It can be understood that the second blind spot warning signal refers to the blind spot warning signal corresponding to the vehicle turning state.

[0100] In a specific implementation, when the current vehicle state is a vehicle turning state, it indicates that the danger level of the traffic target currently traveling in the blind spot is high, and a high-level alarm method is adopted, that is, a second blind spot alarm signal is generated, for example, an icon display plus an alarm prompt sound.

[0101] It should be noted that the blind spot monitoring technology flow chart in this embodiment includes: S1, camera internal and external parameter calibration. In step S1, the camera calibration adopts a fisheye camera model, and uses the calibration site of the surround camera to calibrate the internal and external parameters. At the same time, five cameras are calibrated to obtain the camera internal parameters, distortion parameters and external parameter matrix, that is, the conversion relationship between the five cameras and the vehicle body coordinate system; S2, image acquisition and preprocessing. In step S2, the video stream is transmitted using the camera driver and controller, and the format of the original video stream is converted through a decoding algorithm. Afterwards, the three images of the front, left front and right front are dedistorted and stitched from a bird's-eye view, and the stitched image is used for blind spot monitoring at the front of the vehicle; S3, target detection and tracking. In step S3, the target detection and tracking method based on deep learning is as follows: S301, driving a real car equipped with five cameras on the road, while capturing continuous frame image data, and then annotating traffic targets such as vehicles, pedestrians, and bicycles as deep learning samples; S302, designing a deep model framework for target detection, and using data samples to train the model to obtain a detection model with better performance; S303, converting the model and deploying it on the vehicle-side controller, connecting to the video stream for target detection and outputting category labels and target rectangular boxes; S304, based on the target detection results , use Kalman filtering and Hungarian matching algorithm to track the target; S4, target speed measurement and ranging, step S4, includes two ranging models; S401, the top-view ranging model combines the top-view transformation scale information, and the ranging is linearly related to the number of pixels; S402, the rear-view ranging model uses the geometric relationship between the camera external parameters and the ground constraints for ranging; S403, combined with the results of target tracking and target positioning, as well as time information, calculate the speed of the target relative to the vehicle; S5, blind spot direction alarm, in step S5, the alarm logic is set for different blind spot ranges and the vehicle state. When there is a target in the blind spot in the straight state, it is set to a first-level alarm (only the instrument icon displays the warning), and when there is a turn signal, it is set to a second-level alarm (icon display plus alarm prompt sound). In addition, when the vehicle turns on the right turn signal, the right blind spot monitoring range is expanded to 4.5 meters horizontally in real time, which helps to improve the safety of right turn driving.

[0102] This embodiment obtains a target stitching image by preprocessing the camera monitoring image; obtains a target tracking result based on the target stitching image, the target detection model and the preset target tracking algorithm; inputs the target stitching image into the target ranging model to obtain a target positioning result; and determines the blind spot monitoring result based on the target tracking result and the target positioning result. By calibrating the internal and external parameters of the camera, obtaining the camera monitoring image and preprocessing it, and then tracking and ranging the target, and finally obtaining the blind spot monitoring result, it is achieved that the efficiency of blind spot monitoring is improved while reducing the computing power consumption of the vehicle-side controller.

[0103] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 In the vehicle blind spot monitoring method, step S10 further includes steps S11 to S13:

[0104] Step S11, decoding the camera monitoring image to obtain a decoded camera monitoring image;

[0105] It can be understood that the images captured by the five blind spot monitoring cameras are decoded to obtain decoded camera monitoring images for subsequent image processing. For example, the original image data is obtained from the camera. These data may be unprocessed RAW format data or compressed data in JPEG, H.264 and other formats. Therefore, it is necessary to decode the images captured by the monitoring cameras for subsequent image processing operations.

[0106] Step S12, correcting the decoded camera monitoring image according to the camera intrinsic parameters and the distortion parameters to obtain a corrected monitoring image;

[0107] It can be understood that the corrected monitoring image refers to the corrected blind spot monitoring image.

[0108] In a specific implementation, image correction is performed on the decoded camera monitoring image according to the distortion parameters which are the parameters describing the imaging characteristics of the camera, thereby obtaining a corrected blind spot monitoring image. The decoded camera monitoring image is image corrected in the above manner so that the corrected blind spot monitoring image can be spliced.

[0109] Step S13, performing bird's-eye view stitching on the calibrated monitoring images according to the extrinsic parameter matrix to obtain a target stitching image.

[0110] In a specific implementation, the corrected blind spot monitoring image is stitched according to the position and direction of the camera relative to the world coordinate system, thereby obtaining a stitched image obtained by stitching multiple images from a bird's-eye view.

[0111] It should be noted that the video stream is transmitted by the camera driver and controller, and the format of the original video stream is converted by the decoding algorithm. After that, the three images of the front, left front and right front are dedistorted and stitched from top to bottom, and the stitched image is used for blind spot monitoring at the front of the vehicle.

[0112] In a feasible implementation, step S30 may include steps S31 to S33:

[0113] Step S31, inputting the target stitching image into the overhead distance measurement model in the target distance measurement model according to the overhead transformation scale information to obtain the overhead distance measurement information;

[0114] It can be understood that the bird's-eye view transformation scale information refers to the actual physical size information represented by each pixel in the image when performing the bird's-eye view transformation, and the bird's-eye view distance measurement model refers to a pre-set model for calculating the bird's-eye view distance.

[0115] In a specific implementation, a stitched image obtained by stitching multiple images from a bird's-eye view is input into a bird's-eye view ranging model in a target ranging model, so that the bird's-eye view ranging model combines the actual physical size information represented by each pixel in the image during the bird's-eye view transformation, and performs ranging in a linear relationship with the number of pixels, and finally obtains the bird's-eye view ranging information.

[0116] Step S32, inputting the target stitching image into the rearview ranging model in the target ranging model according to the external parameter matrix and the ground constraint relationship to obtain rearview ranging information;

[0117] It can be understood that the ground constraint relationship refers to some basic principles and constraints that need to be followed in order to accurately map the ground elements in the image to the top-down perspective when performing a bird's-eye view transformation, and the rear-view ranging model refers to a pre-set model for calculating the rear-view distance.

[0118] In a specific implementation, a stitched image obtained by stitching multiple images from a bird's-eye view is input into a rear-view ranging model in a target ranging model, so that the rear-view ranging model combines the position and direction of the camera relative to the world coordinate system and some basic principles and constraints that need to be followed when performing a bird's-eye view transformation in order to accurately map the ground elements in the image to the bird's-eye view to perform ranging, and finally obtains rear-view ranging information.

[0119] Step S33, obtaining a target positioning result according to the overhead distance measurement information and the rearward distance measurement information.

[0120] In a specific implementation, the overhead distance measurement information and the rear-view distance measurement information are summarized, and the positioning information of the vehicle is calculated, and then the target positioning result is obtained according to the calculation result.

[0121] In this embodiment, the target stitching image is input into the overhead distance measurement model in the target distance measurement model according to the overhead transformation scale information to obtain the overhead distance measurement information; the target stitching image is input into the rear-view distance measurement model in the target distance measurement model according to the external parameter matrix and the ground constraint relationship to obtain the rear-view distance measurement information; and the target positioning result is obtained according to the overhead distance measurement information and the rear-view distance measurement information. In the above manner, the positioning efficiency of the blind spot target is improved.

[0122] In a feasible implementation, step S40 may include steps S21 to S22:

[0123] Step S21, inputting the target stitching image into the target detection model to obtain the target detection result;

[0124] It can be understood that the stitched image obtained by stitching multiple images from a bird's-eye view is input into the target detection depth model, and then the image detection result output by the model, that is, the category label and the target rectangular box, is obtained, and finally the target detection result is obtained.

[0125] Step S22, performing target tracking according to the target detection result and a preset target tracking algorithm to obtain a target tracking result.

[0126] It can be understood that the stitched image obtained by inputting multiple images for top-down stitching is input into the target detection depth model to obtain the category label and target rectangular frame output by the model, and then the category label and target rectangular frame output by the model are tracked by a target tracking algorithm such as Kalman filtering or Hungarian matching algorithm, and finally the target tracking result in the vehicle blind spot is obtained.

[0127] In this embodiment, the target detection result is obtained by inputting the target splicing image into the target detection model, and the target tracking result is obtained by performing target tracking according to the target detection result and a preset target tracking algorithm. In this way, the tracking efficiency of the blind spot target is improved.

[0128] This embodiment decodes the camera monitoring image to obtain a decoded camera monitoring image; corrects the decoded camera monitoring image according to the camera internal parameters and distortion parameters to obtain a corrected monitoring image; and performs overhead stitching on the corrected monitoring image according to the external parameter matrix to obtain a target stitching image. Through the above method, the computing power required by the controller for visual blind spot detection is reduced, and a new solution is provided for vehicle blind spot detection.

[0129] For example, in order to help understand the implementation process of the vehicle blind spot monitoring method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5A brief flow chart of a vehicle blind spot monitoring method is provided, specifically: S1, camera internal and external parameter calibration, in the step S1, the camera calibration adopts a fisheye camera model, uses a surround camera calibration site to calibrate the internal and external parameters, and calibrates five cameras at the same time to obtain camera internal parameters, distortion parameters and external parameter matrices, that is, the conversion relationship between the five cameras and the vehicle body coordinate system; S2, image acquisition and preprocessing, in the step S2, the camera driver and controller are used to transmit the video stream, and the original video stream is converted into a format through a decoding algorithm. Afterwards, the three images of the front, left front and right front are dedistorted and stitched from a bird's-eye view, and the stitched image is used for blind spot monitoring at the front of the vehicle; S3, target detection and tracking. In step S3, the target detection and tracking method based on deep learning is as follows: S301, driving a real car equipped with five cameras on the road, while capturing continuous frame image data, and then annotating traffic targets such as vehicles, pedestrians, and bicycles as deep learning samples; S302, designing a deep model framework for target detection, and using data samples to train the model to obtain a detection model with better performance; S303, converting the model and deploying it on the vehicle-side controller, connecting to the video stream for target detection and outputting category labels and target rectangular boxes; S304, based on the target detection results , use Kalman filtering and Hungarian matching algorithm to track the target; S4, target speed measurement and ranging, step S4, includes two ranging models; S401, the top-view ranging model combines the top-view transformation scale information, and the ranging is linearly related to the number of pixels; S402, the rear-view ranging model uses the geometric relationship between the camera external parameters and the ground constraints for ranging; S403, combined with the results of target tracking and target positioning, as well as time information, calculate the speed of the target relative to the vehicle; S5, blind spot direction alarm, in step S5, the alarm logic is set for different blind spot ranges and the vehicle state. When there is a target in the blind spot in the straight state, it is set to a first-level alarm (only the instrument icon displays the warning), and when there is a turn signal, it is set to a second-level alarm (icon display plus alarm prompt sound). In addition, when the vehicle turns on the right turn signal, the right blind spot monitoring range is expanded to 4.5 meters horizontally in real time, which helps to improve the safety of right turn driving.

[0130] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the vehicle blind spot monitoring method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0131] This application also provides a vehicle blind spot monitoring device, please refer to Figure 6 , the vehicle blind spot monitoring device comprises:

[0132] The processing module 10 is used to pre-process the camera monitoring image to obtain a target stitching image;

[0133] A tracking module 20, configured to obtain a target tracking result according to the target spliced ​​image, the target detection model and a preset target tracking algorithm;

[0134] A positioning module 30 is used to input the target stitching image into a target distance measurement model to obtain a target positioning result;

[0135] The monitoring module 40 is used to determine the blind spot monitoring result according to the target tracking result and the target positioning result.

[0136] Optionally, the processing module 10 is further used for:

[0137] Decoding the camera monitoring image to obtain a decoded camera monitoring image;

[0138] Correcting the decoded camera monitoring image according to camera internal parameters and distortion parameters to obtain a corrected monitoring image;

[0139] The corrected monitoring images are stitched from a bird's-eye view according to the external parameter matrix to obtain a target stitched image.

[0140] Optionally, the positioning module 30 is further used for:

[0141] Inputting the target stitching image into the overhead distance measurement model in the target distance measurement model according to the overhead transformation scale information to obtain the overhead distance measurement information;

[0142] Inputting the target stitching image into a backsight ranging model in a target ranging model according to an external parameter matrix and a ground constraint relationship to obtain backsight ranging information;

[0143] A target positioning result is obtained according to the overhead distance measurement information and the rearward distance measurement information.

[0144] Optionally, the tracking module 20 is further configured to:

[0145] Annotate the blind area image data of a preset number of frames to obtain image sample data;

[0146] Training the initial detection model according to the image sample data to obtain a model training result;

[0147] A target detection model is obtained according to the model training results.

[0148] Optionally, the tracking module 20 is further configured to:

[0149] Inputting the target stitching image into the target detection model to obtain the target detection result;

[0150] Target tracking is performed according to the target detection result and a preset target tracking algorithm to obtain a target tracking result.

[0151] Optionally, the processing module 10 is further used for:

[0152] Calibrate the vehicle blind spot cameras at multiple angles according to the preset camera model to obtain the camera intrinsic parameters, distortion parameters and extrinsic parameter matrix;

[0153] Determine a camera coordinate transformation relationship according to the camera intrinsic parameters, the distortion parameters and the extrinsic parameter matrix;

[0154] The calibrated vehicle blind spot cameras at multiple angles are obtained according to the camera coordinate conversion relationship, and the camera monitoring images are collected by the calibrated vehicle blind spot cameras at multiple angles.

[0155] Optionally, the monitoring module 40 is further used for:

[0156] When the blind spot monitoring result is that a traffic target exists, the current vehicle state is obtained;

[0157] When the current vehicle state is a vehicle straight-moving state, generating a first blind spot warning signal;

[0158] When the current vehicle state is a vehicle turning state, a second blind spot warning signal is generated, and the alarm level of the second blind spot warning signal is greater than the alarm level of the first blind spot warning signal.

[0159] The vehicle blind spot monitoring device provided by the present application adopts the vehicle blind spot monitoring method in the above embodiment, which can solve the technical problem of how to improve the monitoring efficiency while reducing the computing power consumption of the vehicle-side controller. Compared with the prior art, the beneficial effects of the vehicle blind spot monitoring device provided by the present application are the same as the beneficial effects of the vehicle blind spot monitoring method provided by the above embodiment, and the other technical features in the vehicle blind spot monitoring device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0160] The present application provides a vehicle blind spot monitoring device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle blind spot monitoring method in the above-mentioned embodiment one.

[0161] Reference below Figure 7, which shows a schematic diagram of the structure of a vehicle blind spot monitoring device suitable for implementing the embodiment of the present application. The vehicle blind spot monitoring device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The vehicle blind spot monitoring device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] like Figure 7 As shown, the vehicle blind spot monitoring device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the vehicle blind spot monitoring device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vehicle blind spot monitoring device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle blind spot monitoring device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0163] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0164] The vehicle blind spot monitoring device provided by the present application adopts the vehicle blind spot monitoring method in the above embodiment, which can solve the technical problem of how to improve the monitoring efficiency while reducing the computing power consumption of the vehicle-side controller. Compared with the prior art, the beneficial effects of the vehicle blind spot monitoring device provided by the present application are the same as the beneficial effects of the vehicle blind spot monitoring method provided by the above embodiment, and the other technical features in the vehicle blind spot monitoring device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0165] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0167] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the vehicle blind spot monitoring method in the above-mentioned embodiment.

[0168] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0169] The computer-readable storage medium may be included in the vehicle blind spot monitoring device; or may exist independently without being assembled into the vehicle blind spot monitoring device.

[0170] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the vehicle blind spot monitoring device, the vehicle blind spot monitoring device: pre-processes the camera monitoring image to obtain a target stitching image; obtains a target tracking result based on the target stitching image, a target detection model and a preset target tracking algorithm; inputs the target stitching image into a target ranging model to obtain a target positioning result; and determines a blind spot monitoring result based on the target tracking result and the target positioning result.

[0171] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0174] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle blind spot monitoring method, and can solve the technical problem of how to improve monitoring efficiency while reducing the computing power consumption of the vehicle-side controller. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the vehicle blind spot monitoring method provided in the above-mentioned embodiment, and will not be repeated here.

[0175] The present application also provides a computer program product, including a computer program, which implements the steps of the vehicle blind spot monitoring method as described above when executed by a processor.

[0176] The computer program product provided by this application can solve the technical problem of how to improve monitoring efficiency while reducing the computing power consumption of the vehicle-side controller. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the vehicle blind spot monitoring method provided by the above embodiment, which will not be repeated here.

[0177] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle blind spot monitoring method, characterized in that: The vehicle blind spot monitoring method comprises: Preprocess the camera monitoring image to obtain the target stitching image; Obtaining a target tracking result according to the target spliced ​​image, the target detection model and the preset target tracking algorithm; Inputting the target stitching image into the target ranging model to obtain a target positioning result; The blind spot monitoring result is determined according to the target tracking result and the target positioning result.

2. The method according to claim 1, characterized in that The preprocessing of the camera monitoring image to obtain the target stitching image includes: Decoding the camera monitoring image to obtain a decoded camera monitoring image; Correcting the decoded camera monitoring image according to camera internal parameters and distortion parameters to obtain a corrected monitoring image; The corrected monitoring images are stitched from a bird's-eye view according to the external parameter matrix to obtain a target stitched image.

3. The method according to claim 1, characterized in that The step of inputting the target stitching image into the target ranging model to obtain the target positioning result includes: Inputting the target stitching image into the overhead distance measurement model in the target distance measurement model according to the overhead transformation scale information to obtain the overhead distance measurement information; Inputting the target stitching image into a backsight ranging model in a target ranging model according to an external parameter matrix and a ground constraint relationship to obtain backsight ranging information; A target positioning result is obtained according to the overhead distance measurement information and the rearward distance measurement information.

4. The method according to claim 1, characterized in that Before obtaining the target tracking result according to the target spliced ​​image, the target detection model and the preset target tracking algorithm, the method further includes: Annotate the blind area image data of a preset number of frames to obtain image sample data; Training the initial detection model according to the image sample data to obtain a model training result; A target detection model is obtained according to the model training results.

5. The method according to claim 1, characterized in that The target tracking result is obtained according to the target stitching image, the target detection model and the preset target tracking algorithm, including: Inputting the target stitching image into the target detection model to obtain the target detection result; Target tracking is performed according to the target detection result and a preset target tracking algorithm to obtain a target tracking result.

6. The method according to claim 1, characterized in that Before the camera monitoring image is preprocessed to obtain the target stitching image, the method further includes: Calibrate the vehicle blind spot cameras at multiple angles according to the preset camera model to obtain the camera intrinsic parameters, distortion parameters and extrinsic parameter matrix; Determine a camera coordinate transformation relationship according to the camera intrinsic parameters, the distortion parameters and the extrinsic parameter matrix; The calibrated vehicle blind spot cameras at multiple angles are obtained according to the camera coordinate conversion relationship, and the camera monitoring images are collected by the calibrated vehicle blind spot cameras at multiple angles.

7. The method according to any one of claims 1 to 6, characterized in that: After determining the blind spot monitoring result according to the target tracking result and the target positioning result, the method further includes: When the blind spot monitoring result is that a traffic target exists, the current vehicle state is obtained; When the current vehicle state is a vehicle straight-moving state, generating a first blind spot warning signal; When the current vehicle state is a vehicle turning state, a second blind spot warning signal is generated, and the alarm level of the second blind spot warning signal is greater than the alarm level of the first blind spot warning signal.

8. A vehicle blind spot monitoring device, characterized in that: The device comprises: A processing module, used for preprocessing the camera monitoring image to obtain a target stitching image; A tracking module, used to obtain a target tracking result according to the target spliced ​​image, the target detection model and a preset target tracking algorithm; A positioning module, used for inputting the target stitching image into the target ranging model to obtain a target positioning result; The monitoring module is used to determine the blind spot monitoring result according to the target tracking result and the target positioning result.

9. A vehicle blind spot monitoring device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle blind spot monitoring method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle blind spot monitoring method according to any one of claims 1 to 7 are implemented.

Images