Low-speed blind area monitoring method and system based on multi-camera

Abstract

The application relates to the technical field of blind area monitoring, and relates to a low-speed blind area monitoring method and system based on multi-view camera shooting, which comprises the following steps: registering multiple cameras in a vehicle to obtain a target vehicle and a vehicle coordinate system; confirming multiple early warning coordinates based on the vehicle coordinate system, a vehicle bird's eye view and the vehicle; confirming a candidate window based on the vehicle bird's eye view and the early warning coordinates; performing target detection on the candidate window by using a pre-constructed target detection model to obtain a detection result; if the detection result is a first result, taking a pre-constructed safety instruction as a target instruction; otherwise, confirming relative positions and relative speeds based on target bounding boxes in a second result and multiple radars in the target vehicle; calculating an expected collision time according to the relative positions and the relative speeds; confirming the target instruction based on the expected collision time; and summarizing the target instruction to obtain multiple target instructions, thereby completing low-speed blind area monitoring. The application can improve the precision of low-speed blind area monitoring.

Description

Technical Field

[0001] This invention relates to the field of blind spot monitoring technology, and in particular to a low-speed blind spot monitoring method and system based on multi-view cameras. Background Technology

[0002] With the development of intelligent driving technology, vehicle safety issues under low-speed conditions have gradually received attention, especially in complex environments such as urban roads, parking lots and construction areas, where drivers find it difficult to fully perceive the surrounding situation of the vehicle through conventional observation methods. As a result, low-speed blind spot monitoring technology has been widely studied and applied.

[0003] Currently, low-speed blind spot monitoring mainly relies on detection methods using a single type of sensor.

[0004] While the methods described above can achieve low-speed blind spot monitoring, the single-sensor approach struggles to balance detection accuracy and stability in various complex environments, making it prone to false alarms or missed detections. Furthermore, the lack of multi-sensor information fusion prevents the comprehensive acquisition of obstacle appearance features, relative position, and velocity information, resulting in insufficient collision prediction accuracy and limiting the practical application of low-speed blind spot monitoring systems. Therefore, improving the accuracy of low-speed blind spot monitoring has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a low-speed blind zone monitoring method based on multi-view cameras and a computer-readable storage medium, the main purpose of which is to improve the accuracy of low-speed blind zone monitoring.

[0006] To achieve the above objectives, the present invention provides a low-speed blind spot monitoring method based on multi-view cameras, comprising:

[0007] Acquire vehicle and speed sensors, wherein the vehicle includes: multiple cameras and multiple radars;

[0008] Multiple cameras in the vehicle are registered to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0009] The speed of the target vehicle is obtained by using a speed sensor to monitor its speed.

[0010] The vehicle speed is compared with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, a bird's-eye view of the vehicle is confirmed based on multiple registered cameras.

[0011] Based on the vehicle coordinate system, the vehicle bird's-eye view, and the confirmed vehicle, multiple warning coordinates were identified.

[0012] For each of the multiple warning coordinates, perform the following operation:

[0013] Candidate windows were identified based on the vehicle's bird's-eye view and warning coordinates;

[0014] The candidate window is detected using a pre-built target detection model to obtain the detection results, which include either a first result or a second result, wherein the second result includes the target bounding box.

[0015] If the detection result is the first result, then the pre-built security instruction will be used as the target instruction;

[0016] Otherwise, the relative position and relative speed are determined based on the target bounding box in the second result and multiple radars in the target vehicle.

[0017] Calculate the estimated collision time based on relative position and relative velocity;

[0018] The target command was confirmed based on the estimated collision time.

[0019] By summarizing the target commands, multiple target commands are obtained, and low-speed blind spot monitoring is completed.

[0020] Optionally, the registration of multiple cameras in the vehicle to obtain the target vehicle and the vehicle coordinate system includes:

[0021] Obtain camera parameters and working plane, where camera parameters include: focal length and sensor height;

[0022] Once the vehicle's center of gravity and longitudinal symmetry center line are identified, the vehicle's center of gravity is projected onto the working plane to obtain the first reference point.

[0023] The vehicle coordinate system is determined based on the longitudinal symmetry centerline and the first reference point;

[0024] The vertical angle of view is calculated based on the focal length and sensor height, using the following formula:

[0025] ,

[0026] in, Indicates a vertical perspective. Indicates the sensor height. Indicates focal length. It is the arctangent function;

[0027] Adjust the pitch angle based on the vertical viewing angle;

[0028] Perform the following operation on each of the multiple cameras in the vehicle:

[0029] By adjusting the pitch angle, the camera angle is adjusted to obtain a registered camera.

[0030] By combining the registered cameras, multiple registered cameras are obtained;

[0031] The target vehicle was identified based on multiple registered cameras.

[0032] Optionally, the process of confirming the vehicle's bird's-eye view based on multiple registered cameras includes:

[0033] Obtain the chessboard calibration plate and debugging cloth;

[0034] Based on the debugging setup, pre-built extrinsic parameter calculation algorithm, and multiple registered cameras, multiple extrinsic parameter matrices were identified;

[0035] Perform the following operation on each of the multiple registration cameras:

[0036] The intrinsic parameters of the registered camera were calibrated using a checkerboard calibration board to obtain the intrinsic parameter matrix and distortion coefficients;

[0037] The original image was confirmed based on the registered camera;

[0038] The distorted image was identified based on the intrinsic parameter matrix, distortion coefficients, and the original image.

[0039] Summarize the distorted images to obtain multiple distorted images;

[0040] The projected image was identified based on multiple extrinsic matrices and multiple distorted images.

[0041] The projected images are fused to obtain a fused image.

[0042] The merged image is cropped to obtain a bird's-eye view of the vehicle.

[0043] Optionally, the step of image fusion of the projected images to obtain a fused image includes:

[0044] Multiple overlapping areas were identified based on the projected images;

[0045] For each of the multiple overlapping regions, perform the following operation:

[0046] Pixels are extracted from the overlapping region to obtain an overlapping pixel value set, which includes red pixel values, green pixel values ​​and blue pixel values.

[0047] For each overlapping pixel value in the overlapping pixel value set, perform the following operation:

[0048] Calculate pixel brightness based on the red, green, and blue pixel values ​​in the overlapping pixel values;

[0049] Sum the pixel brightness to obtain the brightness of multiple pixels;

[0050] The maximum brightness is determined based on the brightness of multiple pixels, where the maximum brightness is the highest image brightness among multiple image brightnesses;

[0051] For each overlapping pixel value in the overlapping pixel value set, perform the following operation:

[0052] The pixel distance is determined based on the overlapping pixel values;

[0053] Calculate pixel confidence based on pixel distance, pixel brightness, and maximum brightness;

[0054] Summarize the pixel confidence scores to obtain multiple pixel confidence scores;

[0055] Calculate the red, green, and blue blended pixel values ​​based on the overlapping pixel value set and multiple pixel confidence levels;

[0056] The merged pixel value is determined based on the red, green, and blue merged pixel values.

[0057] Summarize the merged pixel values ​​to obtain the merged pixel value set;

[0058] The merged image was identified based on the merged pixel value set.

[0059] Optionally, determining the pixel distance based on overlapping pixel values ​​includes:

[0060] The coordinates of overlapping pixels are identified, including the pixel x-coordinate and pixel y-coordinate.

[0061] The principal point coordinates are determined based on the intrinsic parameter matrix of the registered camera corresponding to the overlapping pixel values. The principal point coordinates include the principal point x-coordinate and the principal point y-coordinate.

[0062] The pixel distance is determined based on the pixel x-coordinate, pixel y-coordinate, principal point x-coordinate, and principal point y-coordinate.

[0063] Optionally, the step of determining multiple warning coordinates based on the vehicle coordinate system, the vehicle bird's-eye view, and the vehicle includes:

[0064] Once the driver is identified, the eye point position is determined based on the driver and vehicle coordinate system. The eye point position includes: eye point height, forward distance of eye point, backward distance of eye point, and distance to obstacles at eye point.

[0065] The vehicle was measured to obtain vehicle parameters, including: the height of the front edge of the hood, the width of the A-pillar, and the height of the lower edge of the rear windshield.

[0066] Calculate the blind spot distance based on the eye point height, the forward distance of the eye point, and the height of the front edge of the hood;

[0067] The lateral viewing angle is calculated based on the width of the A-pillar and the distance to the eyepoint obstacle.

[0068] Calculate the rear visibility distance based on the eye point height, the lower edge height of the rear windshield, and the distance behind the eye point.

[0069] The vehicle blind spot map is determined based on the forward blind spot distance, the lateral visibility angle, and the rear visibility distance. The vehicle blind spot map includes multiple vehicle blind spot ranges.

[0070] Mapping multiple vehicle blind spot ranges from the vehicle blind spot map to the vehicle bird's-eye view map yields the blind spot bird's-eye view map, which includes multiple bird's-eye view blind spot ranges.

[0071] For each of the multiple vehicle blind spot ranges, perform the following operation:

[0072] The minimum bounding rectangle is determined based on the vehicle's blind spot range;

[0073] The minimum bounding rectangle is uniformly divided based on a preset sampling window to obtain multiple sampling windows;

[0074] Perform the following operation for each of the multiple sampling windows:

[0075] The coordinates of the window center are determined based on the vehicle coordinate system and the sampling window.

[0076] Sum the center coordinates of the windows to obtain the center coordinates of multiple windows;

[0077] By aggregating the center coordinates of multiple windows, multiple warning coordinates are obtained.

[0078] Optionally, the determination of relative position and relative velocity based on the target bounding box and multiple radars in the target vehicle in the second result includes:

[0079] The center coordinates of the bounding box are determined based on the target bounding box, which include the center x-coordinate and the center y-coordinate.

[0080] Multiple radar coordinates were identified based on multiple radars in the target vehicle. These radar coordinates include the radar x-coordinate and the radar y-coordinate.

[0081] For each of the multiple radar coordinates, perform the following operation:

[0082] Calculate the radar range based on the radar x-coordinate and radar y-coordinate in the radar coordinate system, and the center x-coordinate and center y-coordinate in the bounding box center coordinate system.

[0083] By summing up the radar distances, multiple radar distances are obtained;

[0084] The minimum radar distance was determined based on multiple radar distances, where the minimum radar distance is the smallest among the multiple radar distances;

[0085] The radar corresponding to the minimum radar distance is taken as the target radar;

[0086] The relative position and relative velocity are determined based on the target radar.

[0087] Optionally, the formula for calculating the estimated collision time is as follows:

[0088] ,

[0089] in, Indicates the estimated collision time. Indicates relative position, This indicates relative velocity.

[0090] Optionally, the step of confirming the target command based on the estimated collision time includes:

[0091] Compare the estimated collision time with the preset first time threshold. If the estimated collision time is greater than or equal to the first time threshold, then the pre-constructed first instruction is used as the target instruction.

[0092] Otherwise, compare the collision time with the preset second time threshold. If the expected collision time is greater than or equal to the second time threshold, then use the pre-constructed second instruction as the target instruction.

[0093] Otherwise, the pre-built third instruction will be used as the target instruction.

[0094] To achieve the above objectives, the present invention also provides a low-speed blind spot monitoring system based on multi-view cameras, comprising:

[0095] The target vehicle confirmation module is used to acquire vehicles and speed sensors. The vehicle includes multiple cameras and multiple radars. The module registers the multiple cameras in the vehicle to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0096] The warning coordinate confirmation module compares the vehicle speed with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, it confirms the vehicle bird's-eye view based on multiple registered cameras and confirms multiple warning coordinates based on the vehicle coordinate system, the vehicle bird's-eye view, and the vehicle.

[0097] The target detection and processing module is used to monitor the speed of the target vehicle using a speed sensor to obtain the vehicle speed. For each of the multiple warning coordinates, the following operations are performed: based on the vehicle bird's-eye view and the warning coordinates, candidate windows are identified, and a pre-built target detection model is used to detect targets in the candidate windows to obtain detection results. The detection results include either a first result or a second result. The second result includes the target bounding box. If the detection result is the first result, the pre-built safety instruction is used as the target instruction.

[0098] The blind spot monitoring adjustment module is used to determine the relative position and relative speed based on the target bounding box in the second result and multiple radars in the target vehicle, calculate the estimated collision time based on the relative position and relative speed, determine the target command based on the estimated collision time, summarize the target commands, obtain multiple target commands, and complete the low-speed blind spot monitoring.

[0099] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0100] Memory, storing at least one instruction; and

[0101] The processor executes the instructions stored in the memory to implement the low-speed blind spot monitoring method based on multi-view cameras described above.

[0102] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned low-speed blind spot monitoring method based on multi-view cameras.

[0103] To address the problems described in the background art, this invention acquires information about a vehicle and its speed sensor. The vehicle includes multiple cameras and multiple radars. This provides a material basis for subsequently using the cameras to acquire images of the vehicle's surroundings and the radars to monitor obstacles. Furthermore, the multiple cameras within the vehicle are registered to obtain the target vehicle and its coordinate system. The target vehicle includes multiple registered cameras and multiple radars. This invention improves the accuracy of low-speed blind spot detection by accurately capturing images of the vehicle's surroundings through the registration of the multiple cameras, and allows for speed monitoring of the target vehicle using a speed sensor. The vehicle speed is measured, and it can be seen that this embodiment of the invention uses a speed sensor to monitor the vehicle speed, facilitating subsequent speed comparison. The vehicle speed is compared with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, a bird's-eye view of the vehicle is confirmed based on multiple registered cameras. This embodiment of the invention only activates the registered cameras to capture images of the vehicle's surrounding environment when the vehicle speed is less than or equal to the preset speed threshold, thereby confirming the vehicle's bird's-eye view. Based on the vehicle coordinate system, the vehicle's bird's-eye view, and the vehicle, multiple warning coordinates are identified. This embodiment of the invention improves the accuracy of low-speed blind spot monitoring by analyzing and processing the vehicle's bird's-eye view to obtain the coordinates of multiple key locations requiring focused monitoring, thus improving the accuracy of multiple warning coordinates. Each warning coordinate in the system performs the following operations: Candidate windows are identified based on the vehicle's bird's-eye view and the warning coordinates. This embodiment of the invention identifies candidate windows using warning coordinates, providing a precise monitoring window for subsequent target detection and improving the accuracy of low-speed blind spot monitoring. A pre-built target detection model is used to detect targets within the candidate windows, yielding detection results. These results include a first result and a second result, where the second result includes the target bounding box. This embodiment of the invention monitors candidate windows using a target detection model, obtaining detection results and creating conditions for subsequent detection of obstacle speed and position based on the detection results. If the detection result is the first result, the pre-built safety instruction is used as the target instruction; otherwise, based on... The second result shows that the target bounding box and multiple radars in the target vehicle confirm the relative position and relative speed. It is evident that this embodiment of the invention improves radar utilization by determining whether to activate radar for obstacle position and speed monitoring based on target detection results. The estimated collision time is calculated based on the relative position and relative speed, and the target command is confirmed based on the estimated collision time. This embodiment of the invention provides scientific decision-making instructions for low-speed blind spot monitoring by calculating the estimated collision time and confirming the corresponding target command based on the estimated collision time. Multiple target commands are then aggregated to complete low-speed blind spot monitoring. Therefore, this embodiment of the invention provides separate warnings for obstacles in the blind spot when the vehicle is running at low speed by using multiple target commands.This invention improves the accuracy of low-speed blind zone monitoring. Attached Figure Description

[0104] Figure 1 This is a flowchart illustrating a low-speed blind spot monitoring method based on multi-view cameras according to an embodiment of the present invention.

[0105] Figure 2 This is a functional block diagram of a low-speed blind spot monitoring system based on multi-view cameras provided in an embodiment of the present invention;

[0106] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the low-speed blind spot monitoring method based on multi-view cameras, according to an embodiment of the present invention.

[0107] Explanation of reference numerals in the attached figures:

[0108] 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0109] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0110] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0111] This application provides a low-speed blind spot monitoring method based on multi-view cameras. The executing entity of the low-speed blind spot monitoring method based on multi-view cameras includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the low-speed blind spot monitoring method based on multi-view cameras can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0112] Reference Figure 1 The diagram shown is a flowchart illustrating a low-speed blind spot monitoring method based on multi-view cameras according to an embodiment of the present invention. In this embodiment, the low-speed blind spot monitoring method based on multi-view cameras includes:

[0113] S1. Acquire vehicle and speed sensors, wherein the vehicle includes: multiple cameras and multiple radars.

[0114] For example, Xiao Zhang is an employee at a large vehicle manufacturing plant who needs to perform blind spot monitoring on large vehicles. Therefore, Xiao Zhang obtains information about the vehicles and speed sensors to facilitate subsequent camera registration and blind spot monitoring based on vehicle speed.

[0115] It should be explained that the vehicle is a large automobile, optionally using a FAW JH6 heavy truck; the speed sensor is a vehicle speed sensor, optionally using a ZF Electronics GS101201 speed sensor; the camera is an automotive camera, optionally using a Sodima HD night vision waterproof AHD blind spot vehicle camera; and the radar is a collision avoidance radar, optionally using a Tiejiangjun 3169 radar.

[0116] S2. Register multiple cameras in the vehicle to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0117] Specifically, the registration of multiple cameras in the vehicle to obtain the target vehicle and its coordinate system includes:

[0118] Obtain camera parameters and working plane, where camera parameters include: focal length and sensor height;

[0119] Once the vehicle's center of gravity and longitudinal symmetry center line are identified, the vehicle's center of gravity is projected onto the working plane to obtain the first reference point.

[0120] The vehicle coordinate system is determined based on the longitudinal symmetry centerline and the first reference point;

[0121] The vertical angle of view is calculated based on the focal length and sensor height, using the following formula:

[0122] ,

[0123] in, Indicates a vertical perspective. Indicates the sensor height. Indicates focal length. It is the arctangent function;

[0124] The pitch angle is calculated and adjusted based on the vertical viewing angle, as shown in the following formula:

[0125] ,

[0126] in, This indicates adjusting the pitch angle. This indicates the preset installation height. Indicates the preset nearest visible distance;

[0127] Perform the following operation on each of the multiple cameras in the vehicle:

[0128] By adjusting the pitch angle, the camera angle is adjusted to obtain a registered camera.

[0129] By combining the registered cameras, multiple registered cameras are obtained;

[0130] The target vehicle was identified based on multiple registered cameras.

[0131] It should be explained that camera parameters are a set of values ​​describing the imaging characteristics and spatial geometric relationships of the camera. These parameters include: focal length and sensor height. Focal length refers to the distance from the optical center of the camera lens to the focal point where parallel light converges. Sensor height refers to the vertical dimension of the camera's photosensitive element. The working plane refers to a horizontal plane where the vehicle is parked. The vehicle's center of gravity is the average point of action of all the vehicle's mass distribution, representing the concentrated location of the vehicle's weight. The longitudinal symmetry centerline is the axis of symmetry along the vehicle's longitudinal direction. Projecting the vehicle's center of gravity onto the working plane means projecting the vehicle's center of gravity onto the working plane to obtain a projection point. This projection point is the first reference point. Establishing the vehicle coordinate system based on the longitudinal symmetry centerline and the first reference point means establishing a three-dimensional coordinate system with the first reference point as the origin, the vehicle's forward direction as the positive X-axis, the vehicle's rightward direction as the positive Y-axis, and the vertically upward direction as the positive Z-axis.

[0132] It should be understood that the vertical viewing angle reflects the range that the camera can capture vertically, and its value is usually expressed in degrees (°). The larger the vertical viewing angle, the larger the range that the camera can capture vertically. The nearest point of view (NPV) refers to the horizontal distance between the ground projection point directly below the camera and the point on the ground corresponding to the lowest point of the camera's field of view. Since the front of large vehicles has a certain length, it can obstruct the driver's forward view; therefore, the NPV is usually manually set based on the length of the vehicle's front end. For example, when the front end of a large vehicle is 1.25 meters long, the NPV can be set to 1.25 meters. The installation height is a value manually set by the staff of the large vehicle manufacturing plant based on the vehicle's height. For example, if the vehicle height is 2.5 meters, the installation height is 2.5 meters. Adjusting the pitch angle refers to the angle by which the camera rotates up and down around its horizontal axis. Adjusting the camera's pitch angle to the target pitch angle means adjusting the camera's pitch angle to the adjusted pitch angle. A registered camera refers to a camera whose pitch angle is the adjusted pitch angle. The process of identifying the target vehicle based on multiple registered cameras refers to adjusting the pitch angle of each camera on the vehicle to the corresponding adjustment pitch angle, thereby obtaining multiple registered cameras installed on the vehicle and identifying the target vehicle. The target vehicle is the vehicle after each camera has been registered.

[0133] S3. Use a speed sensor to monitor the speed of the target vehicle, obtain the vehicle speed, compare the vehicle speed with a preset speed threshold, and if the vehicle speed is less than or equal to the preset speed threshold, then confirm the vehicle's bird's-eye view based on multiple registered cameras.

[0134] It should be explained that the speed monitoring of the target vehicle using a speed sensor refers to using a speed sensor to monitor the vehicle's speed, which is the vehicle's velocity. The technology used to monitor vehicle speed using a speed sensor is existing technology and will not be elaborated upon here. The speed threshold is a value manually set by staff at a large vehicle manufacturing plant; optionally, the speed threshold is 30 kilometers per hour.

[0135] Specifically, the process of confirming the vehicle's bird's-eye view based on multiple registered cameras includes:

[0136] Obtain the chessboard calibration plate and debugging cloth;

[0137] Based on the debugging setup, pre-built extrinsic parameter calculation algorithm, and multiple registered cameras, multiple extrinsic parameter matrices were identified;

[0138] Perform the following operation on each of the multiple registration cameras:

[0139] The intrinsic parameters of the registered camera were calibrated using a checkerboard calibration board to obtain the intrinsic parameter matrix and distortion coefficients;

[0140] The original image was confirmed based on the registered camera;

[0141] The distorted image was identified based on the intrinsic parameter matrix, distortion coefficients, and the original image.

[0142] Summarize the distorted images to obtain multiple distorted images;

[0143] The projected image was identified based on multiple extrinsic matrices and multiple distorted images.

[0144] The projected images are fused to obtain a fused image.

[0145] The merged image is cropped to obtain a bird's-eye view of the vehicle.

[0146] It should be explained that the checkerboard calibration board refers to a flat board printed with regular black and white squares, and the calibration cloth refers to a cloth laid on the ground with a known pattern (checkerboard or dot array, etc.). The process of determining multiple extrinsic matrices based on the calibration cloth, a pre-constructed extrinsic parameter calculation algorithm, and multiple registration cameras involves: laying the calibration cloth on the ground around the vehicle; simultaneously photographing the calibration cloth using multiple registration cameras; and using a pre-constructed extrinsic parameter calculation algorithm to map the calibration points in each registration camera image to known 3D coordinates, solving for the rotation matrix and translation vector of each camera relative to the vehicle coordinate system to obtain the extrinsic parameter matrix. Optionally, the PnP (Perspective-n-Point) series algorithm can be used as the extrinsic parameter calculation algorithm. The process of using the checkerboard calibration board to perform intrinsic parameter calibration on the registration camera involves: placing the checkerboard calibration board at different positions within the registration camera's field of view and taking multiple calibration images; and using the Zhang Zhengyou calibration method based on these calibration images to calibrate the registration camera, obtaining the intrinsic parameter matrix and distortion coefficients. The intrinsic parameter matrix refers to the internal parameters that map camera coordinates (normalized coordinates) to pixel coordinates, while the distortion coefficient reflects the geometric distortion caused by the lens.

[0147] It is understood that the process of identifying the original image based on the registered camera refers to: capturing images of the area around the vehicle using the registered camera, which is the original image. The process of identifying the distorted image based on the intrinsic parameter matrix, distortion coefficients, and the original image refers to: performing geometric correction on the original image using inverse mapping and bilinear interpolation based on the intrinsic parameter matrix and distortion coefficients of the registered camera, which is the distorted image. Furthermore, the technique of performing geometric correction on the original image using inverse mapping and bilinear interpolation based on the intrinsic parameter matrix and distortion coefficients of the registered camera is existing technology and will not be elaborated here. The process of identifying the projected image based on multiple extrinsic parameter matrices and multiple distorted images refers to: projecting each distorted image onto a working plane according to its corresponding extrinsic parameters to generate an image under the working plane, which is the projected image. The method of projecting each distorted image onto the working plane according to its corresponding extrinsic parameters is existing technology and will not be elaborated here. The cropping of the fused image refers to cropping out a preset vehicle area from the fused image. This vehicle area is a value manually set by staff at a large automobile manufacturing plant based on the length and width of a large automobile. For example, if a large automobile is 10 meters long and 5 meters wide, the vehicle area would be a range extending 2.5 meters outward from the vehicle's outline. The vehicle bird's-eye view refers to the fused image after the cropping operation.

[0148] Specifically, the process of image fusion of the projected images to obtain a fused image includes:

[0149] Multiple overlapping areas were identified based on the projected images;

[0150] For each of the multiple overlapping regions, perform the following operation:

[0151] Pixels are extracted from the overlapping region to obtain an overlapping pixel value set, which includes red pixel values, green pixel values ​​and blue pixel values.

[0152] For each overlapping pixel value in the overlapping pixel value set, perform the following operation:

[0153] The pixel brightness is calculated based on the red, green, and blue pixel values ​​in the overlapping pixel values, using the following formula:

[0154] ,

[0155] in, Indicates pixel brightness. Indicates the value of the red pixel. Indicates the green pixel value. Indicates the value of the blue pixel. , and These are the preset first brightness coefficient, second brightness coefficient, and third brightness coefficient, respectively;

[0156] Sum the pixel brightness to obtain the brightness of multiple pixels;

[0157] The maximum brightness is determined based on the brightness of multiple pixels, where the maximum brightness is the highest image brightness among multiple image brightnesses;

[0158] For each overlapping pixel value in the overlapping pixel value set, perform the following operation:

[0159] The pixel distance is determined based on the overlapping pixel values;

[0160] The pixel confidence score is calculated based on pixel distance, pixel brightness, and maximum brightness, using the following formula:

[0161] ,

[0162] in, Indicates pixel confidence. Indicates pixel brightness. Indicates maximum brightness. Indicates pixel distance;

[0163] Summarize the pixel confidence scores to obtain multiple pixel confidence scores;

[0164] Calculate the red, green, and blue blended pixel values ​​based on multiple red pixel values ​​and multiple pixel confidence levels in the overlapping pixel value set;

[0165] The merged pixel value is determined based on the red, green, and blue merged pixel values.

[0166] Summarize the merged pixel values ​​to obtain the merged pixel value set;

[0167] The merged image was identified based on the merged pixel value set.

[0168] It should be explained that identifying multiple overlapping regions based on projected images means: first, mapping each projected image onto the vehicle coordinate system, and then comparing the pixel coverage of each projected image to identify multiple overlapping regions. An overlapping region refers to the area formed when multiple registered cameras project onto the working plane, where the same pixel position is covered by multiple images. Extracting pixels from the overlapping regions means: reading the brightness values ​​of the red, green, and blue channels of each pixel within the overlapping region, combining these values ​​to obtain overlapping pixel values. The set of overlapping pixel values ​​is a collection of multiple overlapping pixel values. Red pixel value refers to the brightness value of the red channel within a pixel, green pixel value refers to the brightness value of the green channel within a pixel, and blue pixel value refers to the brightness value of the blue channel within a pixel. Pixel brightness reflects the brightness of a pixel. The first, second, and third brightness coefficients are values ​​manually set by staff at a large vehicle manufacturing plant: the first brightness coefficient is 0.299, the second is 0.587, and the third is 0.114.

[0169] It should be understood that pixel confidence is used to measure the degree of confidence in a pixel's acceptance in an overlapping region. The higher the pixel confidence, the greater the confidence in a pixel's acceptance in the overlapping region. The calculation of the red, green, and blue blended pixel values ​​based on the overlapping pixel value set and multiple pixel confidences refers to: calculating the red blended pixel value using multiple red pixel values ​​and multiple pixel confidences from the overlapping pixel value set; calculating the green blended pixel value using multiple green pixel values ​​and multiple pixel confidences from the overlapping pixel value set; and calculating the blue blended pixel value using multiple blue pixel values ​​and multiple pixel confidences from the overlapping pixel value set. The formula for calculating the red blended pixel value is as follows: ,in, Indicates the red blended pixel value. This represents the number of red pixel values ​​out of a set of red pixel values. Represents the confidence level of multiple pixels. 1 pixel confidence level Represents the first of multiple red pixel values The red pixel value is a weighted average of all red pixel values ​​and their confidence levels. The methods for calculating the green pixel value using multiple green pixel values ​​and their confidence levels from the overlapping pixel value set, and the methods for calculating the blue pixel value using multiple blue pixel values ​​and their confidence levels from the overlapping pixel value set, are the same as the method for calculating the red pixel value using multiple red pixel values ​​and their confidence levels from the overlapping pixel value set, and will not be repeated here. The green pixel value is the final green pixel value obtained by weighted averaging of all green pixel values ​​and their confidence levels, and the blue pixel value is the final blue pixel value obtained by weighted averaging of all blue pixel values ​​and their confidence levels.

[0170] It is understood that determining the fused pixel value based on the red fused pixel value, green fused pixel value and blue fused pixel value means: combining the red fused pixel value, green fused pixel value and blue fused pixel value to obtain the fused pixel value, which is the combined red fused pixel value, green fused pixel value and blue fused pixel value.

[0171] For example, if the red blended pixel value is 100, the green blended pixel value is 140, and the blue blended pixel value is 170, then the blended pixel value is (100, 140, 170).

[0172] It should be understood that the fused pixel value set refers to a collection of multiple fused pixel values. The process of identifying the fused image based on the fused pixel value set means modifying the red, green, and blue pixel values ​​in each overlapping pixel value to their corresponding red, green, and blue fused pixel values ​​in the fused pixel value set. The projected image after modifying these values ​​is the fused image.

[0173] Specifically, determining the pixel distance based on overlapping pixel values ​​includes:

[0174] The coordinates of overlapping pixels are identified, including the pixel x-coordinate and pixel y-coordinate.

[0175] The principal point coordinates are determined based on the intrinsic parameter matrix of the registered camera corresponding to the overlapping pixel values. The principal point coordinates include the principal point x-coordinate and the principal point y-coordinate.

[0176] The pixel distance is determined based on the pixel x-coordinate, pixel y-coordinate, principal point x-coordinate, and principal point y-coordinate.

[0177] It should be explained that the overlapping pixel coordinates refer to the three-dimensional coordinates obtained by back-projecting the overlapping pixels in the images captured by multiple registered cameras onto the vehicle coordinate system after calibration of intrinsic and extrinsic parameters and coordinate transformation. The pixel x-coordinate refers to the x-coordinate of the overlapping pixel in the vehicle coordinate system, and the pixel y-coordinate refers to the y-coordinate of the overlapping pixel in the vehicle coordinate system. The determination of principal point coordinates based on the intrinsic parameter matrix of the registration camera corresponding to the overlapping pixel values ​​refers to: reading the principal point coordinates from the intrinsic parameter matrix. The principal point coordinates refer to the coordinates of the optical center of the registration camera on the imaging plane obtained from the intrinsic parameter matrix. The principal point x-coordinate refers to the x-coordinate of the optical center of the registration camera on the imaging plane, and the principal point y-coordinate refers to the y-coordinate of the optical center of the registration camera on the imaging plane. The pixel distance refers to the geometric distance between the overlapping pixel points and the optical center of the registration camera. The determination of pixel distance based on pixel x-coordinates, pixel y-coordinates, principal point x-coordinates, and principal point y-coordinates refers to: calculating the pixel distance using the distance formula between two points based on the pixel x-coordinates, pixel y-coordinates, principal point x-coordinates, and principal point y-coordinates. The method of calculating the pixel distance using the distance formula between two points based on the pixel x-coordinates, pixel y-coordinates, principal point x-coordinates, and principal point y-coordinates is existing technology and will not be elaborated here.

[0178] S4. Based on the vehicle coordinate system, vehicle bird's-eye view, and vehicle, multiple warning coordinates are identified. For each of the multiple warning coordinates, the following operation is performed: Candidate windows are identified based on the vehicle bird's-eye view and warning coordinates.

[0179] It should be explained that the "candidate window determination based on vehicle bird's-eye view and warning coordinates" means: based on the vehicle bird's-eye view and warning coordinates, a rectangular area is defined in the bird's-eye view with the warning coordinates as the center, serving as the candidate window. The length of the rectangular area is a preset first side length, and the width is a preset second side length. Both the first and second side lengths are values ​​manually set by staff at large vehicle manufacturing plants; optionally, the first side length is 200 pixels and the second side length is 100 pixels. The candidate window refers to the rectangular area determined with the warning coordinates as the center.

[0180] Specifically, the multiple warning coordinates identified based on the vehicle coordinate system, the vehicle bird's-eye view, and the vehicle include:

[0181] Once the driver is identified, the eye point position is determined based on the driver and vehicle coordinate system. The eye point position includes: eye point height, forward distance of eye point, backward distance of eye point, and distance to obstacles at eye point.

[0182] The vehicle was measured to obtain vehicle parameters, including: the height of the front edge of the hood, the width of the A-pillar, and the height of the lower edge of the rear windshield.

[0183] The blind spot distance is calculated based on the eyepoint height, the forward distance of the eyepoint, and the height of the leading edge of the hood. The calculation formula is as follows:

[0184] ,

[0185] in, Indicates the distance to the blind spot ahead. Eyepoint height, Indicates the height of the leading edge of the hood. Eyepoint forward distance;

[0186] The lateral viewing angle is calculated based on the width of the A-pillar and the distance to the eyepoint obstruction, using the following formula:

[0187] ,

[0188] in, Indicates the lateral viewing angle. Indicates the width of column A. Indicates the distance to the eyepoint obstacle;

[0189] The rear visibility distance is calculated based on the eyepoint height, the lower edge height of the rear windshield, and the distance behind the eyepoint. The calculation formula is as follows:

[0190] ,

[0191] in, Indicates the visible distance behind. Indicates the height of the lower edge of the rear windshield. Indicates the distance behind the eye point;

[0192] The vehicle blind spot map is determined based on the forward blind spot distance, the lateral visibility angle, and the rear visibility distance. The vehicle blind spot map includes multiple vehicle blind spot ranges.

[0193] Mapping multiple vehicle blind spot ranges from the vehicle blind spot map to the vehicle bird's-eye view map yields the blind spot bird's-eye view map, which includes multiple bird's-eye view blind spot ranges.

[0194] For each of the multiple vehicle blind spot ranges, perform the following operation:

[0195] The minimum bounding rectangle is determined based on the vehicle's blind spot range;

[0196] The minimum bounding rectangle is uniformly divided based on a preset sampling window to obtain multiple sampling windows;

[0197] Perform the following operation for each of the multiple sampling windows:

[0198] The coordinates of the window center are determined based on the vehicle coordinate system and the sampling window.

[0199] Summarize the coordinates of the window center to obtain the set of window center coordinates;

[0200] Summarize the set of window center coordinates to obtain multiple sets of window center coordinates;

[0201] By using the center coordinates of each window in a set of multiple window center coordinates as warning coordinates, multiple warning coordinates are obtained.

[0202] It should be explained that "driver" refers to a person who drives a large vehicle. The "eye point position determination based on driver and vehicle coordinate system" means: measuring the driver's standard sitting position in the vehicle coordinate system to obtain the eye point position. The eye point position includes: eye point height, which is the height of the driver's eyes from the ground when maintaining a standard sitting position in the vehicle coordinate system; eye point forward distance, which is the horizontal distance from the driver's eyes to the windshield of the vehicle when maintaining a standard sitting position in the vehicle coordinate system; eye point backward distance, which is the horizontal distance from the driver's eyes to the rear windshield of the vehicle when maintaining a standard sitting position in the vehicle coordinate system; and eye point obstacle distance, which is the horizontal distance from the driver's eyes to vehicle A when maintaining a standard sitting position in the vehicle coordinate system.

[0203] It should be understood that the measurement of the vehicle to obtain vehicle parameters refers to measuring the vehicle's basic parameters, which are the vehicle parameters themselves. These parameters include: the height of the front edge of the hood, the width of the A-pillar, and the height of the lower edge of the rear windshield. The height of the front edge of the hood refers to the height of the front edge of the hood from the ground; the width of the A-pillar refers to the width of the pillars on either side of the windshield; and the height of the lower edge of the rear windshield refers to the height of the lower edge of the rear windshield from the ground. The forward blind spot distance is the shortest distance at which the driver cannot see the ground in front of them. The lateral visibility angle is the angle at which the driver cannot observe due to the A-pillar obstruction. The rear visibility distance is the distance from the driver's eye level to the lower edge of the rear windshield. The process of determining a vehicle blind spot map based on forward blind spot distance, lateral visibility angle, and rear visibility distance involves the following steps: In a top-down coordinate system, using the vehicle's outline as a reference, firstly, the forward blind spot distance is extended longitudinally along the front of the vehicle to obtain a rectangular blind spot. Secondly, with the driver's eye point as the center, fan-shaped visual fields are expanded to the left and right of the vehicle according to the lateral visibility angle; the area outside these fan-shaped areas is the lateral blind spot. Finally, the rear blind spot is drawn by extending longitudinally beyond the rear visibility distance behind the vehicle. The blind spots from the front, sides, and rear are then superimposed to obtain the overall distribution map of the driver's invisible area. This overall distribution map of the driver's invisible area is the vehicle blind spot map. The vehicle blind spot range refers to the invisible area in the vehicle blind spot map formed by the vehicle body structure or the driver's eye point limitations.

[0204] It should be explained that mapping multiple blind spot areas from the vehicle blind spot map to the vehicle bird's-eye view means projecting these blind spot areas into the pixel coordinate system of the bird's-eye view through a coordinate transformation, so that each blind spot area in the vehicle's geometric space is marked as a corresponding region in the bird's-eye view. Specifically, the vehicle blind spot map in the vehicle coordinate system uses the driver's eye point as a reference, representing the invisible space surrounding the vehicle; while the vehicle bird's-eye view is a top-down image generated by fusing multiple registered cameras, described using a two-dimensional pixel grid coordinate system. By establishing the correspondence between the vehicle coordinate system and the bird's-eye view pixel coordinate system, the boundary points of the blind spot areas can be converted one by one into pixels in the bird's-eye view, and the corresponding areas can be marked on the bird's-eye view, ultimately obtaining the blind spot bird's-eye view. The blind spot bird's-eye view refers to a vehicle bird's-eye view that identifies the distribution range of each blind spot.

[0205] It is understood that determining the minimum bounding rectangle based on the vehicle blind spot range means: performing geometric boundary fitting on the vehicle blind spot range to obtain the minimum bounding rectangle that can completely enclose the vehicle blind spot range. The minimum bounding rectangle is a rectangle that can completely enclose the vehicle blind spot range.

[0206] For example, if the length of the minimum bounding rectangle is 20 cm and the width is 10 cm, and the preset sampling window is a 5 cm × 5 cm square, then the minimum bounding rectangle is uniformly divided based on the preset sampling window to obtain 8 sampling windows. Optionally, the sampling window is a 5 cm × 5 cm square.

[0207] It should be explained that the statement "determining the window center coordinates based on the vehicle coordinate system and the sampling window" means: using the vehicle coordinate system as a reference system, determining the coordinates of the intersection of the diagonals of the sampling window in the vehicle coordinate system. The coordinates of the intersection of the diagonals of the sampling window in the vehicle coordinate system are the window center coordinates. The window center coordinate set is a set composed of multiple window center coordinates. The warning coordinates refer to the window center coordinates.

[0208] S5. Use a pre-built target detection model to perform target detection on the candidate window and obtain the detection result. The detection result includes either a first result or a second result. The second result includes the target bounding box. If the detection result is the first result, then the pre-built security instruction is used as the target instruction.

[0209] It should be explained that the main operating principle of the target detection model is as follows: First, the candidate window is normalized to obtain a processing window image. Then, the YOLO target model is used to perform forward propagation on the preprocessed image using the processing window image to obtain the detection result. Finally, non-maximum suppression is applied to the detected bounding boxes to obtain the target bounding boxes. The above process is a publicly available technical solution and will not be elaborated here. The first result refers to the absence of obstacles and bounding boxes, and the second result refers to the presence of obstacles and bounding boxes. The target bounding box refers to the boundary marker obtained by completely enclosing the detected target in a rectangular region on the candidate window. The safety instruction refers to the pre-set backtracking operation instruction used to ensure vehicle safety when the detection model cannot confirm the target or the result is uncertain. The target instruction refers to the instruction generated based on the detected target information, used to prompt or control the specific operation of the vehicle.

[0210] S6. Otherwise, based on the target bounding box in the second result and multiple radars in the target vehicle, the relative position and relative speed are confirmed, and the estimated collision time is calculated based on the relative position and relative speed.

[0211] Specifically, the determination of relative position and relative velocity based on the target bounding box and multiple radars in the target vehicle in the second result includes:

[0212] The center coordinates of the bounding box are determined based on the target bounding box, which include the center x-coordinate and the center y-coordinate.

[0213] Multiple radar coordinates were identified based on multiple radars in the target vehicle. These radar coordinates include the radar x-coordinate and the radar y-coordinate.

[0214] For each of the multiple radar coordinates, perform the following operation:

[0215] The radar range is calculated based on the radar x-coordinate and radar y-coordinate in the radar coordinate system, and the center x-coordinate and center y-coordinate in the bounding box center coordinate system. The calculation formula is as follows:

[0216] ,

[0217] in, Indicates radar distance, Represents the radar x-coordinate, Represents the radar's y-coordinate. Represents the center x-coordinate, Indicates the center y-coordinate;

[0218] By summing up the radar distances, multiple radar distances are obtained;

[0219] The minimum radar distance was determined based on multiple radar distances, where the minimum radar distance is the smallest among the multiple radar distances;

[0220] The radar corresponding to the minimum radar distance is taken as the target radar;

[0221] The relative position and relative velocity are determined based on the target radar.

[0222] It should be explained that determining the center coordinates of the target bounding box based on the target bounding box means: identifying the intersection of the diagonals of the target bounding box; measuring the coordinates of this intersection in the vehicle coordinate system; and obtaining the coordinates of the intersection in the vehicle coordinate system. These coordinates are the center coordinates of the bounding box. The center x-coordinate is the x-coordinate of the intersection of the diagonals in the vehicle coordinate system, and the center y-coordinate is the y-coordinate of the intersection in the vehicle coordinate system. Determining multiple radar coordinates based on multiple radars in the target vehicle means: measuring the coordinates of each radar in the target vehicle in the vehicle coordinate system, obtaining multiple radar coordinates. Radar coordinates refer to the coordinates of the radars in the target vehicle in the vehicle coordinate system. The radar x-coordinate and radar y-coordinate are the x-coordinates and y-coordinates of the radars in the target vehicle in the vehicle coordinate system. Radar distance refers to the distance between the radars in the target vehicle and the intersection of the diagonals of the target bounding box. The target radar refers to the radar corresponding to the minimum radar distance.

[0223] In detail, the formula for calculating the estimated collision time is as follows:

[0224] ,

[0225] in, Indicates the estimated collision time. Indicates relative position, This indicates relative velocity.

[0226] It should be explained that the estimated collision time reflects the time interval between the vehicle and the obstacle at the current speed when they meet or come into contact. The shorter the estimated collision time, the higher the potential collision risk.

[0227] S7. Based on the expected collision time, identify the target command, summarize the target commands, obtain multiple target commands, and complete the low-speed blind spot monitoring.

[0228] Specifically, the command to confirm the target based on the estimated collision time includes:

[0229] Compare the estimated collision time with the preset first time threshold. If the estimated collision time is greater than or equal to the first time threshold, then the pre-constructed first instruction is used as the target instruction.

[0230] Otherwise, compare the collision time with the preset second time threshold. If the expected collision time is greater than or equal to the second time threshold, then use the pre-constructed second instruction as the target instruction.

[0231] Otherwise, the pre-built third instruction will be used as the target instruction.

[0232] It should be explained that the first instruction refers to the instruction with low collision risk, which is used to remind the driver to pay attention to the surrounding environment. It can take the form of a warning sound, the instrument panel icon lighting up, or the central control display of a mild warning message. The second instruction refers to the instruction with medium collision risk, which is used to further prompt the driver to take deceleration or evasive action. It can take the form of enhanced sound prompts, steering wheel vibration, head-up display warning boxes, etc. The third instruction refers to the instruction with high collision risk, which is used for emergency warnings or even direct intervention in vehicle control. It can take the form of continuous alarm sounds, seat vibration, automatic intervention of the braking system, or automatic deceleration control.

[0233] For example, once multiple target instructions are obtained, Xiao Zhang can take corresponding actions on the vehicle according to the multiple target instructions, thereby completing the low-speed blind spot monitoring.

[0234] To address the problems described in the background art, this invention acquires information about a vehicle and its speed sensor. The vehicle includes multiple cameras and multiple radars. This provides a material basis for subsequently using the cameras to acquire images of the vehicle's surroundings and the radars to monitor obstacles. Furthermore, the multiple cameras within the vehicle are registered to obtain the target vehicle and its coordinate system. The target vehicle includes multiple registered cameras and multiple radars. This invention improves the accuracy of low-speed blind spot detection by accurately capturing images of the vehicle's surroundings through the registration of the multiple cameras, and allows for speed monitoring of the target vehicle using a speed sensor. The vehicle speed is measured, and it can be seen that this embodiment of the invention uses a speed sensor to monitor the vehicle speed, facilitating subsequent speed comparison. The vehicle speed is compared with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, a bird's-eye view of the vehicle is confirmed based on multiple registered cameras. This embodiment of the invention only activates the registered cameras to capture images of the vehicle's surrounding environment when the vehicle speed is less than or equal to the preset speed threshold, thereby confirming the vehicle's bird's-eye view. Based on the vehicle coordinate system, the vehicle's bird's-eye view, and the vehicle, multiple warning coordinates are identified. This embodiment of the invention improves the accuracy of low-speed blind spot monitoring by analyzing and processing the vehicle's bird's-eye view to obtain the coordinates of multiple key locations requiring focused monitoring, thus improving the accuracy of multiple warning coordinates. Each warning coordinate in the system performs the following operations: Candidate windows are identified based on the vehicle's bird's-eye view and the warning coordinates. This embodiment of the invention identifies candidate windows using warning coordinates, providing a precise monitoring window for subsequent target detection and improving the accuracy of low-speed blind spot monitoring. A pre-built target detection model is used to detect targets within the candidate windows, yielding detection results. These results include a first result and a second result, where the second result includes the target bounding box. This embodiment of the invention monitors candidate windows using a target detection model, obtaining detection results and creating conditions for subsequent detection of obstacle speed and position based on the detection results. If the detection result is the first result, the pre-built safety instruction is used as the target instruction; otherwise, based on... The second result shows that the target bounding box and multiple radars in the target vehicle confirm the relative position and relative speed. It is evident that this embodiment of the invention improves radar utilization by determining whether to activate radar for obstacle position and speed monitoring based on target detection results. The estimated collision time is calculated based on the relative position and relative speed, and the target command is confirmed based on the estimated collision time. This embodiment of the invention provides scientific decision-making instructions for low-speed blind spot monitoring by calculating the estimated collision time and confirming the corresponding target command based on the estimated collision time. Multiple target commands are then aggregated to complete low-speed blind spot monitoring. Therefore, this embodiment of the invention provides separate warnings for obstacles in the blind spot when the vehicle is running at low speed by using multiple target commands.This invention improves the accuracy of low-speed blind zone monitoring.

[0235] like Figure 2 The diagram shown is a functional block diagram of a low-speed blind spot monitoring system based on multi-view cameras provided in an embodiment of the present invention.

[0236] The low-speed blind spot monitoring system 100 based on multi-view cameras described in this invention can be installed in an electronic device. Depending on the functions implemented, the low-speed blind spot monitoring system 100 may include a target vehicle confirmation module 101, a warning coordinate confirmation module 102, a target detection processing module 103, and a blind spot monitoring adjustment module 104. The module described in this invention can also be called a unit, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0237] The target vehicle confirmation module 101 is used to acquire the vehicle and speed sensor. The vehicle includes multiple cameras and multiple radars. The module registers the multiple cameras in the vehicle to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0238] The warning coordinate confirmation module 102 compares the vehicle speed with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, it confirms the vehicle bird's-eye view based on multiple registered cameras and confirms multiple warning coordinates based on the vehicle coordinate system, the vehicle bird's-eye view, and the vehicle.

[0239] The target detection processing module 103 is used to monitor the speed of the target vehicle using a speed sensor to obtain the vehicle speed. For each of the multiple warning coordinates, the following operations are performed: based on the vehicle bird's-eye view and the warning coordinates, candidate windows are identified, and a pre-built target detection model is used to detect the target in the candidate windows to obtain a detection result. The detection result includes either a first result or a second result. The second result includes a target bounding box. If the detection result is the first result, the pre-built safety instruction is used as the target instruction.

[0240] The blind spot monitoring and adjustment module 104 is used to otherwise determine the relative position and relative speed based on the target bounding box in the second result and multiple radars in the target vehicle, calculate the estimated collision time based on the relative position and relative speed, determine the target command based on the estimated collision time, summarize the target commands, obtain multiple target commands, and complete the low-speed blind spot monitoring.

[0241] In detail, the modules in the low-speed blind spot monitoring system 100 based on multi-view cameras described in this embodiment of the invention employ the same methods as described above. Figure 1The method uses the same techniques as the low-speed blind spot monitoring method based on multi-view cameras described above, and can produce the same technical effects, so it will not be repeated here.

[0242] like Figure 3 The diagram shown is a structural schematic of an electronic device for implementing a low-speed blind spot monitoring method based on multi-view cameras, according to an embodiment of the present invention.

[0243] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a low-speed blind spot monitoring method program based on multi-view cameras.

[0244] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a low-speed blind spot monitoring method program based on multi-camera imaging, but also to temporarily store data that has been output or will be output.

[0245] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a low-speed blind spot monitoring method program based on multi-camera imaging) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0246] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0247] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0248] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0249] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0250] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0251] The low-speed blind spot monitoring method program based on multi-camera imaging stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0252] Acquire vehicle and speed sensors, wherein the vehicle includes: multiple cameras and multiple radars;

[0253] Multiple cameras in the vehicle are registered to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0254] The speed of the target vehicle is obtained by using a speed sensor to monitor its speed.

[0255] The vehicle speed is compared with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, a bird's-eye view of the vehicle is confirmed based on multiple registered cameras.

[0256] Based on the vehicle coordinate system, the vehicle bird's-eye view, and the confirmed vehicle, multiple warning coordinates were identified.

[0257] For each of the multiple warning coordinates, perform the following operation:

[0258] Candidate windows were identified based on the vehicle's bird's-eye view and warning coordinates;

[0259] The candidate window is detected using a pre-built target detection model to obtain the detection results, which include either a first result or a second result, wherein the second result includes the target bounding box.

[0260] If the detection result is the first result, then the pre-built security instruction will be used as the target instruction;

[0261] Otherwise, the relative position and relative speed are determined based on the target bounding box in the second result and multiple radars in the target vehicle.

[0262] Calculate the estimated collision time based on relative position and relative velocity;

[0263] The target command was confirmed based on the estimated collision time.

[0264] By summarizing the target commands, multiple target commands are obtained, and low-speed blind spot monitoring is completed.

[0265] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0266] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0267] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0268] Acquire vehicle and speed sensors, wherein the vehicle includes: multiple cameras and multiple radars;

[0269] Multiple cameras in the vehicle are registered to obtain the target vehicle and the vehicle coordinate system. The target vehicle includes multiple registered cameras and multiple radars.

[0270] The speed of the target vehicle is obtained by using a speed sensor to monitor its speed.

[0271] The vehicle speed is compared with a preset speed threshold. If the vehicle speed is less than or equal to the preset speed threshold, a bird's-eye view of the vehicle is confirmed based on multiple registered cameras.

[0272] Based on the vehicle coordinate system, the vehicle bird's-eye view, and the confirmed vehicle, multiple warning coordinates were identified.

[0273] For each of the multiple warning coordinates, perform the following operation:

[0274] Candidate windows were identified based on the vehicle's bird's-eye view and warning coordinates;

[0275] The candidate window is detected using a pre-built target detection model to obtain the detection results, which include either a first result or a second result, wherein the second result includes the target bounding box.

[0276] If the detection result is the first result, then the pre-built security instruction will be used as the target instruction;

[0277] Otherwise, the relative position and relative speed are determined based on the target bounding box in the second result and multiple radars in the target vehicle.

[0278] Calculate the estimated collision time based on relative position and relative velocity;

[0279] The target command was confirmed based on the estimated collision time.

[0280] By summarizing the target commands, multiple target commands are obtained, and low-speed blind spot monitoring is completed.

[0281] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0282] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0283] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0284] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0285] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A low-speed blind area monitoring method based on multi-view camera, characterized in that, The method comprises: acquiring a vehicle and a speed sensor, wherein the vehicle comprises a plurality of cameras and a plurality of radars; registering the plurality of cameras in the vehicle to obtain a target vehicle and a vehicle coordinate system, wherein the target vehicle comprises a plurality of registered cameras and a plurality of radars; monitoring the speed of the target vehicle by using the speed sensor to obtain a vehicle speed; comparing the vehicle speed with a preset speed threshold, and if the vehicle speed is less than or equal to the preset speed threshold, confirming a vehicle bird's eye view based on the plurality of registered cameras; confirming a plurality of early warning coordinates based on the vehicle coordinate system, the vehicle bird's eye view and the vehicle; wherein the confirming the plurality of early warning coordinates based on the vehicle coordinate system, the vehicle bird's eye view and the vehicle comprises: confirming a driver, and confirming an eye point position based on the driver and the vehicle coordinate system, wherein the eye point position comprises an eye point height, an eye point forward distance, an eye point backward distance and an eye point obstacle distance; measuring the vehicle to obtain vehicle parameters, wherein the vehicle parameters comprise a hood front edge height, an A-pillar width and a rear windshield lower edge height; calculating a front blind area distance according to the eye point height, the eye point forward distance and the hood front edge height; calculating a lateral visible angle according to the A-pillar width and the eye point obstacle distance; calculating a rear visible distance according to the eye point height, the rear windshield lower edge height and the eye point backward distance; confirming a vehicle blind area map based on the front blind area distance, the lateral visible angle and the rear visible distance, wherein the vehicle blind area map comprises a plurality of vehicle blind area ranges; mapping the plurality of vehicle blind area ranges in the vehicle blind area map to the vehicle bird's eye view to obtain a blind area bird's eye view, wherein the blind area bird's eye view comprises a plurality of bird's eye blind area ranges; performing the following operations on each of the plurality of vehicle blind area ranges: confirming a minimum circumscribed rectangle based on the vehicle blind area range; uniformly dividing the minimum circumscribed rectangle based on a preset sampling window to obtain a plurality of sampling windows; performing the following operations on each of the plurality of sampling windows: confirming a window center coordinate based on the vehicle coordinate system and the sampling window; summarizing the window center coordinates to obtain a plurality of window center coordinates; summarizing the plurality of window center coordinates to obtain a plurality of early warning coordinates; performing the following operations on each of the plurality of early warning coordinates: confirming a candidate window based on the vehicle bird's eye view and the early warning coordinate; performing target detection on the candidate window by using a pre-constructed target detection model to obtain a detection result, wherein the detection result comprises a first result or a second result, and the second result comprises a target bounding box; if the detection result is the first result, taking a pre-constructed safety instruction as a target instruction; otherwise, confirming a relative position and a relative speed based on the target bounding box in the second result and the plurality of radars in the target vehicle; calculating a predicted collision time according to the relative position and the relative speed; confirming a target instruction based on the predicted collision time; summarizing the target instructions to obtain a plurality of target instructions, and completing low-speed blind area monitoring. 2.The multi-camera based low-speed blind area monitoring method of claim 1, wherein, The registering the plurality of cameras in the vehicle to obtain a target vehicle and a vehicle coordinate system comprises: acquiring camera parameters and a working plane, wherein the camera parameters comprise a focal length and a sensor height; Confirm the vehicle gravity center and the longitudinal symmetry center line, project the vehicle gravity center onto the working plane to obtain a first reference point; Confirm the vehicle coordinate system based on the longitudinal symmetry center line and the first reference point; Calculate the vertical view angle according to the focal length and the sensor height, and the calculation formula is as follows: wherein, denotes the vertical viewing angle, denotes the sensor height, denotes the focal length, is the arc tangent function; Calculate the adjustment pitch angle according to the vertical view angle; The following operations are performed on each of the plurality of cameras in the vehicle: Angle adjustment is performed on the camera based on the adjustment pitch angle to obtain a registration camera; The registration cameras are summarized to obtain a plurality of registration cameras; Confirm the target vehicle based on the plurality of registration cameras. 3.The multi-view camera based low-speed blind area monitoring method of claim 2, wherein, The confirmation of the vehicle bird's eye view based on the plurality of registration cameras comprises: Obtain a checkerboard calibration board and a debugging cloth; Confirm a plurality of extrinsic parameter matrices based on the debugging cloth, a pre-constructed extrinsic parameter calculation algorithm, and the plurality of registration cameras; The following operations are performed on each of the plurality of registration cameras: Use the checkerboard calibration board to calibrate the intrinsic parameters of the registration camera to obtain an intrinsic parameter matrix and a distortion coefficient; Confirm the original image based on the registration camera; Confirm the de-distorted image based on the intrinsic parameter matrix, the distortion coefficient, and the original image; Summarize the de-distorted images to obtain a plurality of de-distorted images; Confirm the projection picture based on the plurality of extrinsic parameter matrices and the plurality of de-distorted images; Image fusion is performed on the projection picture to obtain a fusion picture; Crop the fusion picture to obtain the vehicle bird's eye view. 4.The multi-view camera based low-speed blind area monitoring method of claim 3, wherein, The image fusion of the projection picture to obtain the fusion picture comprises: Confirm a plurality of overlapping regions based on the projection picture; The following operations are performed on each of the plurality of overlapping regions: Pixel extraction is performed on the overlapping region to obtain an overlapping pixel value set, wherein the overlapping pixel value includes a red pixel value, a green pixel value, and a blue pixel value; The following operations are performed on each of the overlapping pixel values in the overlapping pixel value set: Calculate the pixel brightness according to the red pixel value, the green pixel value, and the blue pixel value in the overlapping pixel value; Summarize the pixel brightness to obtain a plurality of pixel brightnesses; Confirm the maximum brightness based on the plurality of pixel brightnesses, wherein the maximum brightness is the maximum image brightness among the plurality of image brightnesses; The following operations are performed on each of the overlapping pixel values in the overlapping pixel value set: Confirm the pixel distance based on the overlapping pixel value; Calculate the pixel confidence according to the pixel distance, the pixel brightness, and the maximum brightness; Summarize the pixel confidence to obtain a plurality of pixel confidences; Calculate the red fusion pixel value, the green fusion pixel value, and the blue fusion pixel value according to the overlapping pixel value set and the plurality of pixel confidences; Confirm the fusion pixel value based on the red fusion pixel value, the green fusion pixel value, and the blue fusion pixel value; Summarize the fusion pixel value to obtain a fusion pixel value set; Confirm the fusion picture based on the fusion pixel value set. 5.The multi-view camera based low-speed blind area monitoring method of claim 4, wherein, The confirmation of the pixel distance based on the overlapping pixel value comprises: Confirm the overlapping pixel coordinates, wherein the overlapping pixel coordinates include a pixel x coordinate and a pixel y coordinate; Confirm the principal point coordinates based on the intrinsic parameter matrix of the registration camera corresponding to the overlapping pixel value, wherein the principal point coordinates include a principal point x coordinate and a principal point y coordinate; Confirm the pixel distance based on the pixel x coordinate, the pixel y coordinate, the principal point x coordinate, and the principal point y coordinate. 6.The multi-view camera based low-speed blind area monitoring method of claim 5, wherein, The relative position and the relative speed are confirmed based on the target bounding box in the second result and the plurality of radars in the target vehicle, and the confirming includes: a bounding box center coordinate is confirmed according to the target bounding box, wherein the bounding box center coordinate includes a center x coordinate and a center y coordinate; a plurality of radar coordinates are confirmed based on the plurality of radars in the target vehicle, wherein the radar coordinates include a radar x coordinate and a radar y coordinate; the following operations are performed on each of the plurality of radar coordinates: a radar distance is calculated according to the radar x coordinate and the radar y coordinate in the radar coordinate, the center x coordinate and the center y coordinate in the bounding box center coordinate; a plurality of radar distances are obtained by aggregating the radar distances; a minimum radar distance is confirmed based on the plurality of radar distances, wherein the minimum radar distance is the smallest radar distance in the plurality of radar distances; a target radar corresponding to the minimum radar distance is taken as the target radar; the relative position and the relative speed are confirmed based on the target radar. 7.The multi-view camera based low-speed blind area monitoring method of claim 6, wherein, The calculation formula of the predicted collision time is as follows: wherein, represents the predicted time to collision, represents the relative position, represents the relative velocity. 8.The multi-view camera based low-speed blind area monitoring method of claim 7, wherein, The target instruction is confirmed based on the predicted collision time, and the confirming includes: the predicted collision time is compared with a preset first time threshold value, if the predicted collision time is greater than or equal to the first time threshold value, a first instruction pre-constructed is taken as the target instruction; otherwise, the collision time is compared with a preset second time threshold value, if the predicted collision time is greater than or equal to the second time threshold value, a second instruction pre-constructed is taken as the target instruction; otherwise, a third instruction pre-constructed is taken as the target instruction.

9. A low-speed blind area monitoring system based on multi-view camera, characterized in that, The system includes: a target vehicle confirming module configured to acquire a vehicle and a speed sensor, wherein the vehicle includes a plurality of cameras and a plurality of radars, and the plurality of cameras in the vehicle are registered to obtain a target vehicle and a vehicle coordinate system, wherein the target vehicle includes a plurality of registered cameras and a plurality of radars; The early warning coordinate confirmation module is configured to monitor the speed of the target vehicle by using the speed sensor to obtain the vehicle speed, compare the vehicle speed with the preset speed threshold, and if the vehicle speed is less than or equal to the preset speed threshold, confirm the vehicle bird's eye view based on the plurality of registration cameras, and confirm the plurality of early warning coordinates based on the vehicle coordinate system, the vehicle bird's eye view and the vehicle. The confirmation of the plurality of early warning coordinates based on the vehicle coordinate system, the vehicle bird's eye view and the vehicle includes confirming the driver, confirming the eye point position based on the driver and the vehicle coordinate system, wherein the eye point position includes the eye point height, the eye point forward distance, the eye point backward distance and the eye point obstacle distance, measuring the vehicle to obtain the vehicle parameters, wherein the vehicle parameters include the engine hood front edge height, the A-pillar width and the rear windshield lower edge height, calculating the front blind area distance according to the eye point height, the eye point forward distance and the engine hood front edge height, calculating the lateral visibility angle according to the A-pillar width and the eye point obstacle distance, calculating the rear visibility distance according to the eye point height, the rear windshield lower edge height and the eye point backward distance, confirming the vehicle blind area map based on the front blind area distance, the lateral visibility angle and the rear visibility distance, wherein the vehicle blind area map includes a plurality of vehicle blind area ranges, mapping the plurality of vehicle blind area ranges in the vehicle blind area map to obtain the blind area bird's eye view, wherein the blind area bird's eye view includes a plurality of bird's eye blind area ranges, and performing the following operations on each of the plurality of vehicle blind area ranges: confirming the minimum circumscribed rectangle based on the vehicle blind area range, uniformly dividing the minimum circumscribed rectangle based on the preset sampling window to obtain a plurality of sampling windows, and performing the following operations on each of the plurality of sampling windows: confirming the window center coordinate based on the vehicle coordinate system and the sampling window, collecting the window center coordinates to obtain a plurality of window center coordinates, and collecting the plurality of window center coordinates to obtain the plurality of early warning coordinates. The target detection processing module is configured to perform the following operations on each of the plurality of early warning coordinates: confirming the candidate window based on the vehicle bird's eye view and the early warning coordinate, performing target detection on the candidate window by using the pre-constructed target detection model to obtain the detection result, wherein the detection result includes the first result or the second result, and the second result includes the target bounding box. If the detection result is the first result, the pre-constructed safety instruction is taken as the target instruction. The blind area monitoring and adjusting module is configured to confirm the relative position and the relative speed based on the target bounding box in the second result and the plurality of radars in the target vehicle, calculate the predicted collision time according to the relative position and the relative speed, confirm the target instruction based on the predicted collision time, collect the target instructions to obtain a plurality of target instructions, and complete the low-speed blind area monitoring.

Images