Vehicle collision warning and processing method and device based on ultrasound and 3D surround view

Abstract

The present invention provides a vehicle collision warning and processing method and device based on ultrasound and 3D surround view. The method comprises: acquiring ultrasonic signal data, including ultrasonic reflection signals emitted by the vehicle and reflected by obstacles; filtering the ultrasonic reflection signals; obtaining a distance warning level based on a preset first judgment model; outputting a warning signal to the driver in response to the identification of an obstacle within a preset warning area, determining the obstacle's direction, and obtaining obstacle direction information; acquiring raw image data around the vehicle and establishing a 3D surround view model of the vehicle; and switching to a 3D surround view virtual camera perspective that matches the obstacle direction information. This invention enables automatic switching of the 3D surround view virtual camera perspective based on obstacle direction, improving vehicle driving safety.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle assistance systems, and in particular to a vehicle collision warning and processing method and device based on ultrasound and 3D surround vision. Background Art

[0002] The current car collision system is mainly used for reversing, and the image is activated only when reversing. The image content includes rear-view image and panoramic overhead image content. However, there are defects in the detection and display of obstacles in other directions, and the panoramic image in this mode cannot fully display the three-dimensional obstacles around the vehicle.

[0003] Although 3D surround-view images can solve the problem of poor display of three-dimensional obstacles around the vehicle, the driver needs to actively start and switch the perspective, and it cannot provide early warning to the driver, posing a safety hazard.

[0004] Therefore, there is an urgent need to propose a car collision warning method and device based on ultrasound and 3D surround vision to solve the technical problem in the existing technology that the viewing angle cannot be automatically switched according to the location of the obstacle. Summary of the Invention

[0005] In view of this, it is necessary to provide a car collision warning method and device based on ultrasound and 3D surround vision to solve the technical problem in the existing technology that the viewing angle cannot be automatically switched according to the location of the obstacle.

[0006] In one aspect, the present invention provides a vehicle collision warning method based on ultrasound and 3D surround vision, comprising:

[0007] Acquiring ultrasonic signal data, the ultrasonic signal data including ultrasonic reflection signals emitted by the vehicle and reflected by an obstacle;

[0008] Filtering the ultrasonic reflection signal;

[0009] Performing a warning distance judgment on the filtered ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level;

[0010] In response to the recognition of an obstacle in a preset warning area, a warning signal is output to the driver according to the distance warning level, and the obstacle direction is determined based on a preset second judgment model to obtain obstacle direction information;

[0011] Obtain the original image data around the vehicle and build a 3D surround view model of the vehicle;

[0012] Switch to a 3D surround view virtual camera perspective that matches the obstacle direction information.

[0013] In a possible implementation, the filtering algorithm for filtering the ultrasonic reflection signal is a Kalman filtering algorithm.

[0014] In one possible implementation, the distance warning level corresponds to the area where the obstacle is located, and the area is divided into a safe area, a warning area, and an alarm area from far to close to the vehicle;

[0015] Among them, the early warning area includes the warning area and the alarm area.

[0016] In a possible implementation, both the first judgment model and the second judgment model adopt neural network models.

[0017] In a possible implementation, before acquiring the raw image data around the vehicle, the following steps are further included:

[0018] Calculate the camera's intrinsic parameters and distortion coefficients according to Zhang Zhengyou's calibration method to perform image distortion correction;

[0019] Based on the unified world coordinate system method, the coordinates of the corner points of the ground calibration object are obtained, the perspective transformation matrix is ​​calculated, and the top view of the calibration object is switched;

[0020] Calculate the weights of the pixels in the image using the weighted average method, and create a corresponding weighted image based on the top view;

[0021] Get the exposure compensation parameters of the top view;

[0022] Calculate the vertex data of the 3D surround stereo model and the texture coordinate data of the texture map;

[0023] Sets the virtual camera's viewpoint to match the preset obstacle orientation.

[0024] In a possible implementation, when obstacles exist in at least two obstacle directions, the virtual camera viewing angle is a 360° panoramic viewing angle.

[0025] In one possible implementation, obtaining the corner coordinates of a ground calibration object based on the unified world coordinate system method includes:

[0026] Place multiple parallel black square calibration objects at the corners of the vehicle, and the black square calibration objects are within the shooting range of the vehicle camera;

[0027] Get the target image containing the black square calibration object;

[0028] The target image is binarized to obtain a binary image in which the calibration object area is white;

[0029] Screen the calibration object area according to the preset constraints and determine the calibration object contour map;

[0030] Perform Hough linear transformation on the calibration object contour image to obtain the length and position of the calibration object straight line and determine the edge of the calibration object;

[0031] Fit the mathematical formula of the black square calibration line and calculate the coordinates of the corner points of the calibration object;

[0032] The sub-pixel corner point precision method is used to optimize the corner point coordinates.

[0033] In one possible implementation, obtaining raw image data around the vehicle and establishing a 3D surround view model of the vehicle includes:

[0034] The original image of the vehicle's surrounding environment is obtained based on the on-board fisheye camera arranged around the vehicle, and the original image is subjected to distortion correction and top view conversion processing;

[0035] Perform fusion processing of the top view to obtain a 2D bird's-eye view;

[0036] The ellipsoid model and the plane model are used as the stereo model of the 3D surround view, and the 2D bird's-eye view is set as the texture map of the stereo model;

[0037] The texture coordinate data is calculated according to the perspective relationship between the three-dimensional model and the plane model, and the texture coordinate data is normalized.

[0038] In a possible implementation, before acquiring the ultrasonic signal data, the following steps are further included:

[0039] Obtaining ultrasonic signal data samples;

[0040] Generate a neural network pre-training model based on ultrasonic signal data samples.

[0041] In another aspect, the present invention further provides a vehicle collision warning device based on an ultrasonic and 3D surround view vehicle collision warning method, comprising:

[0042] A first acquisition module is used to acquire ultrasonic signal data, where the ultrasonic signal data includes ultrasonic reflection signals emitted by the vehicle and reflected by obstacles;

[0043] A filtering module, used for filtering the ultrasonic reflection signal;

[0044] A judgment module, configured to perform a warning distance judgment on the filtered ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level;

[0045] a warning module, configured to, in response to recognition of an obstacle within a preset warning area, output a warning signal to the driver based on the distance warning level, and determine the direction of the obstacle based on a preset second judgment model to obtain obstacle direction information;

[0046] The second acquisition module is used to acquire the original image data around the vehicle and establish a 3D surround view model of the vehicle;

[0047] The processing module is used to switch to a 3D surround view virtual camera perspective that matches the obstacle direction information.

[0048] The beneficial effects of adopting the above embodiment are: after obtaining ultrasonic signal data, the ultrasonic reflection signal is filtered; judgment is made based on a preset judgment model to obtain the obstacle warning level and obstacle direction information, and corresponding to the identification of the obstacle, a warning signal is output to the driver; then a 3D surround stereo model of the vehicle is established, and a 3D surround virtual camera perspective that matches the obstacle direction information is obtained. When obstacles are detected around the vehicle, the perspective is switched to a 3D surround virtual camera perspective that matches the direction of the obstacle, without the driver having to manually switch, thereby improving the safety of the vehicle during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A schematic flow chart of an embodiment of a vehicle collision warning method based on ultrasound and 3D surround vision provided by the present invention;

[0051] Figure 2 A flowchart of an embodiment of a preset method before acquiring ultrasonic signal data provided by the present invention;

[0052] Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of step S201;

[0053] Figure 4 For the present invention Figure 1 A schematic flow chart of an embodiment of step S105;

[0054] Figure 5 A schematic diagram of a scene in which ultrasound and 3D surround view are fused as provided by the present invention;

[0055] Figure 6 This is a schematic diagram of the optimization of the intersection of the 3D surround view stereo model provided by the present invention;

[0056] Figure 7 A schematic diagram of generating texture coordinate data provided by the present invention;

[0057] Figure 8 A schematic structural diagram of an embodiment of a vehicle collision warning device based on ultrasound and 3D surround vision provided by the present invention;

[0058] Figure 9 This is a schematic structural diagram of an embodiment of a vehicle collision warning device based on ultrasound and 3D surround vision provided by the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0060] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0061] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] The embodiments of the present invention provide a vehicle collision warning and processing method and device based on ultrasound and 3D surround vision, which are described below respectively.

[0063] Figure 1 A schematic flow chart of an embodiment of a vehicle collision warning method based on ultrasound and 3D surround vision provided by the present invention; Figure 2 A flowchart of an embodiment of a preset method before acquiring ultrasonic signal data provided by the present invention; Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of step S201; Figure 4 For the present invention Figure 1 FIG. 5 is a flow chart of an embodiment of step S105 in FIG.

[0064] like Figure 1 As shown, the vehicle collision warning method based on ultrasound and 3D surround view includes:

[0065] S101, acquiring ultrasonic signal data, where the ultrasonic signal data includes ultrasonic reflection signals emitted by a vehicle and reflected by an obstacle;

[0066] S102, filtering the ultrasonic reflection signal;

[0067] S103, performing a warning distance judgment on the filtered ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level;

[0068] S104: In response to recognition of an obstacle within the preset warning area, output a warning signal to the driver according to the distance warning level, and determine the direction of the obstacle based on a preset second judgment model to obtain obstacle direction information;

[0069] S105, acquiring original image data around the vehicle and establishing a 3D surround view model of the vehicle;

[0070] S106: Switch to a 3D surround view virtual camera perspective that matches the obstacle direction information.

[0071] Among them, after receiving the ultrasonic reflection signal emitted by the vehicle and reflected by the obstacle, the ultrasonic reflection signal is filtered to achieve a rapid response to the obstacle; the distance of the filtered ultrasonic reflection signal is judged based on the preset first judgment model to obtain the warning level; when an obstacle is identified in the preset warning area, a warning signal is output to the driver according to the preset warning level, and the direction of the obstacle is judged; then, based on the acquisition of the original image data around the vehicle, a 3D surround view model of the vehicle is built, and the 3D surround view virtual camera perspective that matches the obstacle direction information is switched. The 3D surround view virtual camera perspective can be automatically switched according to the obstacle direction, thereby improving the safety of the vehicle during driving.

[0072] Compared with the existing technology, the automobile collision warning method based on ultrasound and 3D surround vision provided by the embodiment of the present invention, through the fusion of ultrasound and 3D surround vision, automatically switches to a virtual perspective corresponding to the direction of the obstacle when an obstacle is detected in the warning area, which can effectively avoid the occurrence of collision accidents due to driving blind spots.

[0073] In one embodiment, in S102, the filtering algorithm for filtering the ultrasonic reflection signal is a Kalman filtering algorithm, and the formula is:

[0074]

[0075]

[0076] in, is the prior estimate, is the current estimated value, is the prior measurement value, for The error covariance matrix, P t for The error covariance matrix, R is the covariance matrix of the measurement noise, and T is the measurement value z t and the prior estimate The distance threshold is , and α and β are coefficients.

[0077] It should be noted that, compared with the existing Kalman algorithm, the present invention adopts an improved Kalman filter algorithm, which can achieve a rapid response to a sudden obstacle and a delayed response to an obstacle that suddenly leaves the measurable range.

[0078] Furthermore, during the ultrasonic ranging process, a sliding window method is used to read data.

[0079] In one embodiment, a warning distance judgment is performed on the ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level. The algorithm of the preset first judgment model here adopts a neural network algorithm.

[0080] It is understandable that the corresponding warning level is different for different distances between the obstacle and the vehicle. Generally, the farther the distance between the obstacle and the vehicle, the lower the distance warning level; the closer the distance between the obstacle and the vehicle, the higher the distance warning level.

[0081] In one embodiment, the distance warning level corresponds to the area where the obstacle is located. The area where the obstacle is located is divided into a safety area, a warning area, and an alarm area from far to near the vehicle. For example, in one scenario, the area 2 meters away from the vehicle is set as a safety area, the area between 1 meter and 2 meters away from the vehicle is set as a warning area, and the area within 1 meter away from the vehicle is set as an alarm area. The examples here are only used to understand the technical solution and are not intended to limit the technical solution. The distance range can be flexibly set according to the needs of the actual scenario.

[0082] In S104, the preset early warning area includes a warning area and an alarm area. In response to the identification of obstacles in the early warning area, a warning signal is output to the driver according to the distance warning level. Specifically, the higher the distance warning level, the greater the intensity and frequency of the warning signal. At the same time, the obstacle direction is judged based on the preset second judgment model, and the direction information of the obstacle is obtained. Usually, in order to achieve all-round monitoring of the vehicle's surrounding environment, fisheye cameras and ultrasonic waves need to be installed on all sides of the vehicle, so that after obtaining the direction information of the obstacle, the corresponding image data and virtual camera perspective are retrieved.

[0083] Furthermore, the obstacle direction information here includes the obstacle distance warning level, obstacle direction, and whether obstacles exist in multiple directions at the same time.

[0084] Figure 5 A schematic diagram of a scene in which ultrasound and 3D surround vision are fused, as provided by the present invention. Figure 5 As shown, the vehicle's obstacle directions are set to multiple directions, V1 to V6. When obstacles exist in at least two directions of the vehicle, the corresponding 3D surround virtual perspective is a 360-degree panoramic perspective. In one embodiment, after establishing the vehicle's 3D surround view model, the 3D surround view virtual camera perspective is switched to match the obstacle direction information. When the preset obstacle distance level is higher, the obstacle size in the 3D surround view is larger, thereby achieving the purpose of warning the driver.

[0085] In one embodiment, if Figure 5 As shown, the ultrasonic sensors for receiving ultrasonic reflection signals are arranged in front of the front of the vehicle, at the left and right corners of the front of the vehicle, above the front wheels, in front of the rear of the vehicle, at the left and right corners of the rear of the vehicle, at the corners of the rear wheels, and above the rear wheels. The radial angle of the ultrasonic sensor is not greater than 75°, and the time consumed for collecting ultrasonic sensor data is less than 70ms, which can effectively improve the judgment accuracy and computing efficiency.

[0086] like Figure 2 As shown, before obtaining the original image data around the vehicle, the following steps are also included:

[0087] S201, calculating the camera intrinsic parameters and distortion coefficients according to the Zhang Zhengyou calibration method, and performing image distortion correction;

[0088] S202: Obtain the corner coordinates of the ground calibration object based on the unified world coordinate system method, calculate the perspective transformation matrix, and switch the top view of the calibration object.

[0089] In one embodiment, if Figure 5 As shown, the calibration objects are four black square papers, which are set at the four corners of the vehicle, so that the calibration objects are all located within the shooting range of the fisheye camera on the vehicle body. The calibration objects are set parallel to each other to facilitate the determination of the world coordinates of the corner points.

[0090] S203 , calculating the weights of the pixels in the image according to a weighted average method, and establishing a corresponding weighted image based on the top view to improve the operation speed.

[0091] S204, obtaining exposure compensation parameters of the top view;

[0092] It is understood that the top view here refers to the top view of the orientation of the calibration object. The exposure compensation parameter of the top view is calculated to solve the problem of uneven brightness in the overlapping area caused by different exposure levels of the fisheye camera during shooting. The formula is:

[0093]

[0094] Among them, g i and g j Represent the gain coefficients of image i and image j respectively, N ij Indicates the number of pixels in the overlapping area of ​​image i and image j, I ij (I ji ) represents the average intensity of image i(j) in the overlapping area, σ N and σ g They represent the standard deviation of error and gain, respectively. Each top view corresponds to one equation.

[0095] S205 : Calculate vertex data of the 3D surround-view stereo model and texture coordinate data of the texture map.

[0096] Specifically, the vertex coordinate calculation formula of the three-dimensional model is:

[0097]

[0098] Generate vertices in a specific α angle range to obtain a surface, set the y-axis coordinates of the vertices at the bottom to be the same to obtain a plane, and set a vertex p on the plane to p To other vertices p i The slope of the vector is p i The slope f i , and then modify the y-axis coordinates of some vertices at the intersection to modify the slope f i , and the further away from p p The slope becomes larger until it approaches a vertex p on the surface. q The slope f q , take the vertex p i (i=j) Two vertices with the same number of vertices are separated as p p and p q , the number of interval vertices is recorded as N, then p p is pi (i = jN-1), p q is pi (i=j+N+1), first calculate p q The slope f q , and then divide by 2n+2 to get the slope Δf of each increment q , the formula is:

[0099]

[0100] Figure 6 This is a schematic diagram of the optimization of the intersection of the 3D surround stereo model provided by the present invention.

[0101] Modify p i (i=jN) to p i The y-axis coordinate of (i=j+N) is used to obtain the vertex coordinates of the final three-dimensional model. The process of optimizing the intersection is as follows: Figure 6 As shown, the formula is:

[0102]

[0103] Figure 7 This is a schematic diagram of generating texture coordinate data provided by the present invention.

[0104] Calculate the texture coordinate data, as shown in the attached Figure 7 As shown, the formula is:

[0105]

[0106] Among them, (x q ,y q ,z q ) represents the coordinates of the surface vertex, (x p ,0,z p ) represents the coordinates of the plane vertex, (x t ,0,z t ) represents the texture coordinates, (0,y c ,0) represents the coordinates of the overlooking viewpoint.

[0107] x t and z t The two-dimensional data are normalized separately, and the normalization range is 0 to 1 to facilitate texture mapping rendering.

[0108] S206: Setting a virtual camera viewing angle that matches a preset obstacle direction.

[0109] In an embodiment of the present invention, the camera internal parameters and distortion coefficients are obtained through the Zhang Zhengyou calibration method to perform image distortion correction, and the corner coordinates of the ground calibration object are obtained based on the unified world coordinate system method. The perspective transformation matrix is ​​calculated to achieve the bird's-eye view switching of the orientation of the calibration object. Then, the weights of the pixel points in the image are calculated according to the weighted average method, and the corresponding weighted image is established to improve the calculation speed. After obtaining the exposure parameters of the bird's-eye view, the 3D surround view model is inspected and textured, and a virtual camera perspective is set to match the preset obstacle direction, which can realize the rapid construction of a 3D surround view stereo model.

[0110] In one embodiment, both the first judgment model and the second judgment model adopt neural network models.

[0111] The neural pre-training models of the first judgment model and the second judgment model are prepared by the following method:

[0112] Five temporally continuous groups of ultrasonic signal data are taken as a sample, that is, one sample consists of 80 data. The input layer has 80 neurons and the ultrasonic signal data are labeled according to the classification. The output layer in the obstacle distance classification model has 3 neurons, and the output layer in the obstacle direction classification model has 7 neurons. There are two middle layers with 20 neurons each. The activation function of the middle layer is leaky relu, the activation function of the output layer is softmax, and the error function is the cross entropy function.

[0113] In one embodiment, when there are obstacles in at least two obstacle directions, the virtual camera perspective is a 360° panoramic perspective. When there is an obstacle in only one obstacle direction, the corresponding virtual camera perspective is the perspective corresponding to the obstacle direction, that is, the display range of the three-dimensional model displayed by the virtual camera perspective at this time includes the direction where the obstacle is located.

[0114] Figure 3 For the present invention Figure 2 A flow chart of an embodiment of step S201 in FIG. Figure 3 As shown in the figure, the corner coordinates of the ground calibration object obtained based on the unified world coordinate system method include:

[0115] S301, placing a plurality of mutually parallel black square calibration objects at the corners of the vehicle, wherein the black square calibration objects are within the shooting range of the vehicle-mounted camera;

[0116] S302, obtaining a target image including a black square calibration object;

[0117] S303, binarizing the target image to obtain a binary image in which the calibration object area is white;

[0118] S304, screening the calibration object area according to the preset constraint conditions to determine the calibration object contour map;

[0119] S305, performing Hough linear transformation on the calibration object contour image to obtain the length and position of the calibration object straight line and determine the edge of the calibration object;

[0120] S306, fitting a mathematical formula for the straight line of the black square calibration object, and calculating the coordinates of the corner points of the calibration object;

[0121] S307: Optimize the corner point coordinates using a sub-pixel corner point precision method.

[0122] Figure 4 For the present invention Figure 1 A flow chart of an embodiment of step S105 in FIG. Figure 4, obtaining the original image data around the vehicle and building the vehicle's 3D surround view model include:

[0123] S401, acquiring an original image of the vehicle's surrounding environment based on a vehicle-mounted fisheye camera disposed around the vehicle, and performing distortion correction and top view conversion processing on the original image;

[0124] S402, performing fusion processing of the top view to obtain a 2D bird's-eye view;

[0125] S403, using the ellipsoid model and the plane model as a 3D surround stereo model, and setting the 2D bird's-eye view as a texture map of the stereo model;

[0126] S404: Calculate texture coordinate data according to the perspective relationship between the three-dimensional model and the two-dimensional model, and perform normalization processing on the texture coordinate data.

[0127] Specifically, in the process of building a 3D surround stereo model, first, the vertex data, texture coordinate data, and memory space for texture mapping data of the stereo model are clamped, and then the ultrasonic signal data is blocked and waited for, corresponding to the acquisition of obstacle information, and the corresponding virtual camera perspective is set; then the pixel coordinates of the undistorted image, the lens intrinsic parameters and the distortion coefficient are used to calculate the coordinates of the distorted image, and then the pixel values ​​are calculated using the interpolation algorithm and assigned to the undistorted image to realize the correction of the fisheye camera image; finally, the perspective transformation matrix is ​​used to transform the original image into a top view, and then multiplied by the respective exposure compensation parameters to realize exposure compensation, the pixel value of the image is multiplied by the pixel value corresponding to the authority value, and the transformed top view is image fused to obtain a 2D bird's-eye view and update the texture image parameters, which can realize the rapid and accurate establishment of the 3D surround stereo model.

[0128] In one embodiment, before acquiring the ultrasonic signal dataset, the method further includes:

[0129] Obtaining ultrasonic signal data samples;

[0130] Generate a neural network pre-training model based on ultrasonic signal data samples.

[0131] Figure 8 This is a schematic structural diagram of an embodiment of a vehicle collision warning device based on ultrasound and 3D surround vision provided by the present invention.

[0132] In order to better implement the vehicle collision warning method based on ultrasound and 3D surround view in the embodiment of the present invention, on the basis of the vehicle collision warning method based on ultrasound and 3D surround view, the corresponding embodiment of the present invention also provides a vehicle collision warning device based on ultrasound and 3D surround view, such as Figure 8 Shown, including:

[0133] A first acquisition module 801 is configured to acquire ultrasonic signal data, where the ultrasonic signal data includes ultrasonic reflection signals emitted by a vehicle and reflected by an obstacle;

[0134] The filtering module 802 is used to filter the ultrasonic reflection signal;

[0135] A judgment module 803 is configured to determine a warning distance of the ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level;

[0136] Warning module 804, configured to, in response to identification of an obstacle within a preset warning area, output a warning signal to the driver based on the distance warning level, and determine the direction of the obstacle based on a preset second judgment model to obtain obstacle direction information;

[0137] The second acquisition module 805 is used to acquire original image data around the vehicle and establish a 3D surround view model of the vehicle;

[0138] The processing module 806 is configured to switch to a 3D surround view virtual camera perspective that matches the obstacle direction information.

[0139] The above embodiment provides a vehicle collision warning device based on ultrasound and 3D surround view, which can implement the technical solution described in the above embodiment of the vehicle collision warning method based on ultrasound and 3D surround view. The specific implementation principles of the above modules or units can be found in the corresponding content of the embodiment of the vehicle collision warning method based on ultrasound and 3D surround view, which will not be repeated here.

[0140] like Figure 9 As shown, the present invention also provides a vehicle collision warning device 900 based on ultrasound and 3D surround view. The vehicle collision warning device 900 based on ultrasound and 3D surround view includes a processor 901, a memory 902 and a display 903. Figure 9 Only some components of the ultrasonic and 3D surround view based vehicle collision warning device 900 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components may be implemented instead.

[0141] In some embodiments, the memory 902 may be an internal storage unit of the ultrasonic and 3D surround view vehicle collision warning device 900, such as a hard disk or memory of the ultrasonic and 3D surround view vehicle collision warning device 900. In other embodiments, the memory 902 may also be an external storage device of the ultrasonic and 3D surround view vehicle collision warning device 900, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the ultrasonic and 3D surround view vehicle collision warning device 900.

[0142] Furthermore, the memory 902 may include both an internal storage unit of the ultrasonic and 3D surround view vehicle collision warning device and an external storage device. The memory 902 is used to store application software and various data installed in the ultrasonic and 3D surround view vehicle collision warning device 900.

[0143] In some embodiments, the processor 901 can be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run the program code or process data stored in the memory 902, such as the ultrasonic and 3D surround view vehicle collision warning method in the present invention.

[0144] In some embodiments, the display 903 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 903 is used to display information on the ultrasonic and 3D surround view vehicle collision warning device 900 and to display a visual user interface. Components 601-603 of the ultrasonic and 3D surround view vehicle collision warning device 900 communicate with each other via a system bus.

[0145] In some embodiments of the present invention, when the processor 901 executes the ultrasonic and 3D surround view vehicle collision warning method stored in the memory 902, the following steps may be implemented:

[0146] Acquiring ultrasonic signal data, the ultrasonic signal data including ultrasonic reflection signals emitted by the vehicle and reflected by an obstacle;

[0147] Filtering the ultrasonic reflection signal;

[0148] Performing a warning distance judgment on the ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level;

[0149] In response to the recognition of an obstacle in a preset warning area, a warning signal is output to the driver according to the distance warning level, and the obstacle direction is determined based on a preset second judgment model to obtain obstacle direction information;

[0150] Obtain the original image data around the vehicle and build a 3D surround view model of the vehicle;

[0151] Switch to a 3D surround view virtual camera perspective that matches the obstacle direction information. It should be understood that when the processor 901 executes the ultrasonic and 3D surround view vehicle collision warning program in the memory 902, in addition to the above functions, it can also implement other functions. For details, please refer to the description of the corresponding method embodiment above.

[0152] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions of the ultrasonic and 3D surround view vehicle collision warning method provided in the above-mentioned method embodiments can be implemented.

[0153] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0154] The above is a detailed introduction to the vehicle collision warning and processing method and device based on ultrasound and 3D surround view provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A vehicle collision warning and processing method based on ultrasound and 3D surround view, characterized in that: include: Acquiring ultrasonic signal data, the ultrasonic signal data including ultrasonic reflection signals emitted by the vehicle and reflected by an obstacle; performing filtering processing on the ultrasonic reflection signal; Performing a warning distance judgment on the filtered ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level; In response to recognition of an obstacle within a preset warning area, outputting a warning signal to the driver according to the distance warning level, and determining the direction of the obstacle based on a preset second judgment model to obtain obstacle direction information; Acquire raw image data around the vehicle and establish a 3D surround view model of the vehicle; Switching to a 3D surround view virtual camera perspective that matches the obstacle direction information; The acquiring of raw image data around the vehicle and establishing a 3D surround view model of the vehicle includes: Acquiring an original image of the vehicle's surrounding environment based on a vehicle-mounted fisheye camera disposed around the vehicle, and performing distortion correction and top view conversion processing on the original image; Performing fusion processing on the top view to obtain a 2D bird's-eye view; The ellipsoid model and the plane model are used as a 3D surround stereo model, and the 2D bird's-eye view is set as a texture map of the stereo model; Texture coordinate data is calculated according to the perspective relationship between the three-dimensional model and the plane model, and the texture coordinate data is normalized.

2. The vehicle collision warning and processing method according to claim 1, characterized in that: The filtering algorithm for filtering the ultrasonic reflection signal is a Kalman filtering algorithm.

3. The vehicle collision warning and processing method according to claim 1, characterized in that: The distance warning level corresponds to the area where the obstacle is located. The area where the obstacle is located is divided into a safety area, a warning area, and an alarm area from far to near to the vehicle; The early warning area includes the warning area and the alarm area.

4. The vehicle collision warning and processing method according to claim 3, characterized in that: The first judgment model and the second judgment model both adopt neural network models.

5. The vehicle collision warning and processing method according to claim 1, characterized in that: Before obtaining the raw image data around the vehicle, it also includes: Calculate the camera's intrinsic parameters and distortion coefficients according to Zhang Zhengyou's calibration method to perform image distortion correction; Based on the unified world coordinate system method, the coordinates of the corner points of the calibration object are obtained, the perspective transformation matrix is ​​calculated, and the top view of the calibration object is switched; Calculating the weights of pixels in the image according to a weighted average method, and establishing a corresponding weighted image based on the top view; Obtaining exposure compensation parameters of the top view; Calculate the vertex data of the 3D surround stereo model and the texture coordinate data of the texture map; Sets the virtual camera's viewpoint to match the preset obstacle orientation.

6. The vehicle collision warning and processing method according to claim 5, characterized in that: When there are obstacles in at least two of the obstacle directions, the virtual camera viewing angle is a 360° panoramic viewing angle.

7. The vehicle collision warning and processing method according to claim 5, characterized in that: The method of obtaining the corner coordinates of the ground calibration object based on the unified world coordinate system includes: Setting a plurality of mutually parallel black square calibration objects at the corners of the vehicle, wherein the black square calibration objects are within the shooting range of the vehicle-mounted camera; Acquire a target image containing the black square calibration object; Binarizing the target image to obtain a binary image in which the calibration object area is white; Screen the calibration object area according to the preset constraints and determine the calibration object contour map; Performing Hough linear transformation on the calibration object contour image to obtain the length and position of the calibration object straight line and determine the edge of the calibration object; Fitting the mathematical formula of the black square calibration object straight line to calculate the coordinates of the corner points of the calibration object; The corner point coordinates are optimized using a sub-pixel corner point precision method.

8. The vehicle collision warning and processing method according to claim 1, characterized in that: Before acquiring the ultrasonic signal data, the method further includes: Obtaining ultrasonic signal data samples; A neural network pre-training model is generated based on the ultrasonic signal data samples.

9. A vehicle collision warning and processing device based on ultrasound and 3D surround view, applicable to the vehicle collision warning and processing method according to any one of claims 1 to 8, characterized in that: include: a first acquisition module, configured to acquire ultrasonic signal data, wherein the ultrasonic signal data includes ultrasonic reflection signals emitted by the vehicle and reflected by obstacles; A filtering module, configured to filter the ultrasonic reflection signal; A judgment module, configured to perform a warning distance judgment on the ultrasonic reflection signal based on a preset first judgment model to obtain a distance warning level; a warning module, configured to, in response to recognition of an obstacle within a preset warning area, output a warning signal to the driver according to the distance warning level, and determine the direction of the obstacle based on a preset second judgment model to obtain obstacle direction information; A second acquisition module is used to acquire original image data around the vehicle and establish a 3D surround view model of the vehicle; A processing module is used to switch to a 3D surround view virtual camera perspective that matches the obstacle direction information.

Images