Obstacle detection method and device fusing vision and ultrasonic waves

Abstract

The invention provides a vision and ultrasonic wave fused obstacle detection method and device, and relates to the technical field of automatic parking. The method comprises the following steps: acquiring an initial obstacle position and an initial parking space position through visual data acquisition equipment; acquiring an initial obstacle point position through ultrasonic data acquisition equipment; converting the initial obstacle position, the initial parking space position and the initial obstacle point position to a world coordinate system to obtain a middle obstacle position, a target parking space position and a middle obstacle point position; matching the position of the middle obstacle with the position of the middle obstacle point; correcting a middle obstacle point in the middle obstacle by adopting the target parking space position to obtain a target obstacle point in the middle obstacle; and constructing a target obstacle according to the distance between the target obstacle point and the first preset edge and the second preset edge in each target parking space. The accuracy and stability of recognizing the contour position of the column obstacle can be improved, and it is ensured that the vehicle is safely parked in the parking space.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic parking, and in particular to an obstacle detection method and device integrating vision and ultrasonic waves. Background Art

[0002] In parking lots, you will often encounter pillar obstacles, most of which are close to the edge of the parking space, which makes it difficult for the automatic parking system to park safely. In order to avoid collision with pillar obstacles and park safely, the automatic parking system needs to be able to identify the outline of the pillar obstacles next to the target parking space.

[0003] Currently, mass-produced models with automatic parking functions are generally equipped with cameras and ultrasonic sensors, and the detection of pillar obstacles mainly uses visual or ultrasonic perception methods. The visual method is mainly based on a deep learning model, using a 2D frame to detect the outline of the pillar obstacle. Since most pillar obstacles are rectangular, during the parking space search process, the camera will observe the pillar obstacle from different angles. In order to make the 2D frame surround the pillar obstacle, this will cause the 2D frame position to jump, and the outline size detected by the 2D frame is larger than the actual size of the pillar obstacle, which sometimes causes the pillar obstacle to invade the parking space, making the available parking area smaller, resulting in too many parking attempts, affecting parking efficiency. The ultrasonic method mainly determines the size of the pillar obstacle contour by scanning the surface obstacle points on the side of the pillar obstacle facing the ultrasonic probe. Since the corners of the pillar obstacle have a poor reflection effect on ultrasonic waves, the ultrasonic probe often fails to detect the corners of the pillar obstacle accurately, resulting in the obstacle points at the corners of the pillar obstacle sometimes being detected in the parking space, affecting the parking space, or causing the corners of the pillar obstacle to be missed, making the detected pillar obstacle contour smaller than the actual size of the pillar obstacle, which will bring collision risks when parking in the parking space. Therefore, the current method for detecting the contour of the pillar obstacle needs to be optimized to improve the accuracy and stability of identifying the contour position of the pillar obstacle and ensure that the vehicle is safely parked in the parking space. Summary of the invention

[0004] The purpose of the present invention is to provide an obstacle detection method and device that integrates vision and ultrasound to address the deficiencies in the above-mentioned prior art, so as to improve the accuracy and stability of identifying the contour position of the obstacle of the pillar and ensure that the vehicle is safely parked in the parking space.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides an obstacle detection method integrating vision and ultrasound, comprising: obtaining an initial obstacle position and an initial parking space position through a visual data acquisition device; obstacles corresponding to the initial obstacle position include pillar obstacles; obtaining an initial obstacle point position through an ultrasonic data acquisition device; converting the initial obstacle position, the initial parking space position, and the initial obstacle point position to a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; matching the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; using the target parking space position to match the intermediate obstacle point position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point. The intermediate obstacle point in the intermediate obstacle is corrected to obtain a target obstacle point in the intermediate obstacle; the adjacent target parking space of the intermediate obstacle is determined according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space; the target parking space is composed of a third preset side, a first preset side, a fourth preset side, and a second preset side connected in sequence; the first preset side is parallel to the second preset side; based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, a target projection point and a target contour length are determined according to the side position and direction of the adjacent target parking space; and a target obstacle is constructed according to the target projection point and the target contour length.

[0007] In one embodiment, the initial obstacle position, the initial parking space position, and the initial obstacle point position are converted to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position, including: calibrating the internal and external parameters of the visual data acquisition device to obtain a calibration table of the visual data acquisition device; according to the calibration table, converting the initial obstacle position and the initial parking space position to the first obstacle position and the first parking space position in the target vehicle coordinate system; according to the relationship between the characteristic object of the target vehicle in the time-synchronized positioning data and the world coordinate system, converting the first obstacle position, the first parking space position, and the initial obstacle point position to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

[0008] In one embodiment, the matching of the intermediate obstacle position with the intermediate obstacle point position to obtain the correspondence between the intermediate obstacle and the intermediate obstacle point includes: determining a first distance between each of the intermediate obstacle points and each of the intermediate obstacles; determining whether a minimum first distance is less than a preset threshold, and when the minimum first distance is less than the preset threshold, determining that the intermediate obstacle point corresponding to the minimum first distance belongs to the intermediate obstacle.

[0009] In one embodiment, determining the first distance between each of the intermediate obstacle points and each of the intermediate obstacles includes: when a projection point of the intermediate obstacle point on a line segment formed by connecting two edge grounding points of the intermediate obstacle is located within the line segment, determining the first distance based on a projection distance on the line segment of a vector connecting the intermediate obstacle point and any one of the two edge grounding points; when a projection point of the intermediate obstacle point on the line segment formed by connecting two edge grounding points of the intermediate obstacle is located outside the line segment, determining the distance between the intermediate obstacle point and each of the two edge grounding points, and determining the minimum distance as the first distance.

[0010] In one embodiment, the target parking space position is used to correct the intermediate obstacle point in the intermediate obstacle to obtain the target obstacle point in the intermediate obstacle, including: determining the projection distance of a vector formed by connecting each corner point of the third preset side in each target parking space and each intermediate obstacle point on the third preset side to obtain a second distance and a third distance; the third preset side is perpendicular to the second preset side; if the product of the second distance and the third distance is a positive number, it indicates that the intermediate obstacle point is located outside the target parking space; if the product of the second distance and the third distance is a negative number, it indicates that the intermediate obstacle point is located in the target parking space; remove the intermediate obstacle point located in the target parking space to obtain the target obstacle point in the intermediate obstacle.

[0011] In one embodiment, the determining of the adjacent target parking spaces of the intermediate obstacle based on the distances between the target obstacle point and the first preset side and the second preset side in each target parking space includes: determining the absolute minimum second distance and the absolute minimum third distance, and if the absolute minimum second distance is a positive number, the target parking spaces are divided into the first group; otherwise, the target parking spaces are divided into the second group; the absolute minimum second distance is the absolute minimum second distance between the target obstacle point and the second preset side in the target parking space; the absolute minimum third distance is the absolute minimum third distance between the target obstacle point and the first preset side in the target parking space ; If the first group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the first group are both positive numbers, and the absolute minimum third distance is smaller than the absolute minimum second distance, then the target parking space corresponding to the absolute minimum third distance is determined as the adjacent target parking space of the intermediate obstacle; if the second group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the second group are both negative numbers, and the absolute minimum second distance is smaller than the absolute minimum third distance, then the target parking space corresponding to the absolute minimum second distance is determined as the adjacent target parking space of the intermediate obstacle.

[0012] In one embodiment, based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, the target projection point and the target contour length are determined according to the side position and direction of the adjacent target parking space; and the target obstacle is constructed according to the target projection point and the target contour length, including: when there are adjacent target parking spaces on both sides of the intermediate obstacle, the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on one side is determined to obtain a fourth distance; the target obstacle point corresponding to the maximum fourth distance is determined, and the projection point of the target obstacle point corresponding to the maximum fourth distance on the second preset side is used as the first target projection point; the projection point of the target obstacle point corresponding to the maximum fourth distance on the first preset side in the adjacent target parking space on the other side is used as the second target projection point; the distance between the first target projection point and the second target projection point is used as the target contour length; and the target obstacle formed by the target contour is determined according to the target contour length, the first target projection point, the second target projection point, the second preset side direction in the adjacent target parking space on one side, and the first preset side direction in the adjacent target parking space on the other side.

[0013] In one embodiment, based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, the target projection point and the target contour length are determined according to the edge position and direction of the adjacent target parking space; the target obstacle is constructed according to the target projection point and the target contour length, including: when the adjacent target parking space exists on one side of the intermediate obstacle, the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset edge in the adjacent target parking space is determined on the second preset edge to obtain a fifth distance; the target obstacle point corresponding to the maximum fifth distance is determined, and the projection point of the target obstacle point corresponding to the maximum fifth distance on the second preset edge is used as the third target projection point; the projection distance of the vector formed by connecting each target obstacle point with the third target projection point on the third preset edge is determined to obtain a sixth distance, and the sixth distance with the maximum absolute value is used as the target contour length; the target obstacle formed by the target contour is determined according to the third target projection point, the target contour length, the third preset edge direction, and the second preset edge direction in the adjacent target parking space.

[0014] In one embodiment, the visual data acquisition device includes a camera, and the ultrasonic data acquisition device includes an ultrasonic radar; 4 of the visual data acquisition devices and 12 of the ultrasonic data acquisition devices are installed on the target vehicle, and the 4 visual data acquisition devices are installed in the middle of the front and rear of the vehicle, and at the positions of the left and right rearview mirrors; the 12 ultrasonic data acquisition devices include 8 ultrasonic UPA probes and 4 ultrasonic APA probes, and 4 of the 8 ultrasonic UPA probes are installed at the front and rear of the vehicle, and 1 of the 4 ultrasonic APA probes is installed on both sides of the front and rear of the vehicle.

[0015] In a second aspect, an embodiment of the present application further provides an obstacle detection device integrating vision and ultrasound, comprising: an acquisition module, configured to acquire an initial obstacle position and an initial parking space position through a visual data acquisition device; obstacles corresponding to the initial obstacle position include pillar obstacles; an initial obstacle point position is acquired through an ultrasonic data acquisition device; a coordinate conversion module, configured to convert the initial obstacle position, the initial parking space position, and the initial obstacle point position into a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; a matching module, configured to match the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; a correction module, configured to adopt the The target parking space position corrects the intermediate obstacle point in the intermediate obstacle to obtain the target obstacle point in the intermediate obstacle; the first determination module is configured to determine the adjacent target parking space of the intermediate obstacle according to the distance between the target obstacle point and the first preset edge and the second preset edge in each target parking space; the target parking space is composed of a third preset edge, a first preset edge, a fourth preset edge, and a second preset edge connected in sequence; the first preset edge is parallel to the second preset edge; the second determination module is configured to determine the target projection point and the target contour length based on the relative position relationship between the intermediate obstacle and the adjacent target parking space and according to the edge position and direction of the adjacent target parking space; the target obstacle is constructed according to the target projection point and the target contour length.

[0016] In one embodiment, the coordinate conversion module is configured to: calibrate the internal and external parameters of the visual data acquisition device to obtain a calibration table of the visual data acquisition device; according to the calibration table, convert the initial obstacle position and the initial parking space position to the first obstacle position and the first parking space position in the target vehicle coordinate system; according to the relationship between the characteristic object of the target vehicle in the time-synchronized positioning data and the world coordinate system, convert the first obstacle position, the first parking space position, and the initial obstacle point position to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

[0017] In one embodiment, the matching module is configured to: determine the first distance between each of the intermediate obstacle points and each of the intermediate obstacles; determine whether the minimum first distance is less than a preset threshold, and when the minimum first distance is less than the preset threshold, determine that the intermediate obstacle point corresponding to the minimum first distance belongs to the intermediate obstacle.

[0018] In one embodiment, the matching module is configured as follows: when the projection point of the intermediate obstacle point on the line segment formed by connecting the two edge grounding points of the intermediate obstacle is located within the line segment, the first distance is determined based on the projection distance of the vector formed by connecting the intermediate obstacle point and any one of the two edge grounding points on the line segment; when the projection point of the intermediate obstacle point on the line segment formed by connecting the two edge grounding points of the intermediate obstacle is located outside the line segment, the distance between the intermediate obstacle point and each of the two edge grounding points is determined, and the minimum distance is determined as the first distance.

[0019] In one embodiment, the correction module is configured to: determine the projection distance of the vector formed by connecting each corner point of the third preset edge in each of the target parking spaces and each of the intermediate obstacle points on the third preset edge to obtain the second distance and the third distance; the third preset edge is perpendicular to the second preset edge; if the product of the second distance and the third distance is a positive number, it indicates that the intermediate obstacle point is located outside the target parking space; if the product of the second distance and the third distance is a negative number, it indicates that the intermediate obstacle point is located inside the target parking space; remove the intermediate obstacle point located inside the target parking space to obtain the target obstacle point in the intermediate obstacle.

[0020] In one embodiment, the first determination module is configured to: determine the absolute minimum second distance and the absolute minimum third distance, and if the absolute minimum second distance is a positive number, the target parking space is divided into the first group; otherwise, the target parking space is divided into the second group; the absolute minimum second distance is the absolute minimum second distance between the target obstacle point and the second preset edge in the target parking space; the absolute minimum third distance is the absolute minimum third distance between the target obstacle point and the first preset edge in the target parking space; if the first group is not empty and the absolute minimum second distance in the first group is positive, the target parking space is divided into the first group; otherwise, the target parking space is divided into the second group; the absolute minimum second distance is the absolute minimum second distance between the target obstacle point and the second preset edge in the target parking space; the absolute minimum third distance is the absolute minimum third distance between the target obstacle point and the first preset edge in the target parking space; if the first group is not empty and the absolute minimum third distance in the first group is positive, the target parking space is divided into the first group; otherwise, the target parking space is divided into the second group; If the absolute minimum second distance and the absolute minimum third distance are both positive numbers, and the absolute minimum third distance is smaller than the absolute minimum second distance, the target parking space corresponding to the absolute minimum third distance is determined as the adjacent target parking space of the intermediate obstacle; if the second group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the second group are both negative numbers, and the absolute minimum second distance is smaller than the absolute minimum third distance, the target parking space corresponding to the absolute minimum second distance is determined as the adjacent target parking space of the intermediate obstacle.

[0021] In one embodiment, the second determination module is configured to: when there are adjacent target parking spaces on both sides of the intermediate obstacle, determine the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on one side on the second preset side to obtain a fourth distance; determine the target obstacle point corresponding to the maximum fourth distance, and use the projection point of the target obstacle point corresponding to the maximum fourth distance on the second preset side as the first target projection point; use the projection point of the target obstacle point corresponding to the maximum fourth distance on the first preset side in the adjacent target parking space on the other side as the second target projection point; use the distance between the first target projection point and the second target projection point as the target contour length; and determine the target obstacle formed by the target contour according to the target contour length, the first target projection point, the second target projection point, the second preset side direction in the adjacent target parking space on one side, and the first preset side direction in the adjacent target parking space on the other side.

[0022] In one embodiment, the second determination module is configured to: when the adjacent target parking space exists on one side of the intermediate obstacle, determine the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on the second preset side to obtain a fifth distance; determine the target obstacle point corresponding to the maximum fifth distance, and use the projection point of the target obstacle point corresponding to the maximum fifth distance on the second preset side as the third target projection point; determine the projection distance of the vector formed by connecting each target obstacle point with the third target projection point on the third preset side to obtain a sixth distance, and use the sixth distance with the maximum absolute value as the target contour length; determine the target obstacle formed by the target contour according to the third target projection point, the target contour length, the third preset side direction, and the second preset side direction in the adjacent target parking space.

[0023] In one embodiment, the visual data acquisition device includes a camera, and the ultrasonic data acquisition device includes an ultrasonic radar; 4 of the visual data acquisition devices and 12 of the ultrasonic data acquisition devices are installed on the target vehicle, and the 4 visual data acquisition devices are installed in the middle of the front and rear of the vehicle, and at the positions of the left and right rearview mirrors; the 12 ultrasonic data acquisition devices include 8 ultrasonic UPA probes and 4 ultrasonic APA probes, and 4 of the 8 ultrasonic UPA probes are installed at the front and rear of the vehicle, and 1 of the 4 ultrasonic APA probes is installed on both sides of the front and rear of the vehicle.

[0024] In a third aspect, an embodiment of the present application further provides a server, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of any one of the above methods.

[0025] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above methods are executed.

[0026] The beneficial effects of the present application are as follows: by acquiring the obstacle position, fusing the obstacle point position, matching the obstacle point position with the obstacle position, and utilizing the position and direction of the edge line of the parking space adjacent to the obstacle, the obstacle contour (i.e., the target obstacle) is finally generated, thereby effectively overcoming the defect of unstable detection results of the pillar obstacle by the visual method, and correcting the pillar obstacle contour based on the parking space position information, thereby improving the accuracy of the ultrasonic method in detecting the position of the pillar obstacle contour, thereby providing strong perception data support for the safe parking of the parking space adjacent to the pillar obstacle, and having great engineering application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the installation of a data acquisition device in a target vehicle in an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0030] Figure 3 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of an initial obstacle detection frame in an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0032] Figure 5 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of the positional relationship between intermediate obstacle points and intermediate obstacles in an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0034] Figure 7 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0035] Figure 8 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of the relative position of an intermediate obstacle point and a target parking space in an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0037] Fig.10 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of target parking space grouping in an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0039] Fig.12 A schematic diagram of constructing a target outline of a target obstacle in a method for obstacle detection that integrates vision and ultrasound provided in an embodiment of the present application;

[0040] Fig.13 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0041] Fig.14 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application;

[0042] Fig.15 A schematic diagram of the structure of an obstacle detection device integrating vision and ultrasound provided in an embodiment of the present application;

[0043] Fig.16 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0045] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0046] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0047] In addition, the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0049] Figure 1 A flowchart of an obstacle detection method integrating vision and ultrasound provided in an embodiment of the present application; Figure 1 As shown, the method includes the following steps 110 to 160:

[0050] Step 110: Acquire the initial obstacle position and the initial parking space position through the visual data acquisition device; and acquire the initial obstacle point position through the ultrasonic data acquisition device.

[0051] Among them, the obstacles corresponding to the initial obstacle position include pillar obstacles; the pillar obstacles mainly refer to the pillar obstacles around the parking space.

[0052] The visual data acquisition device may be a device capable of acquiring images, such as a camera; the ultrasonic data acquisition device may be a device capable of acquiring ultrasonic data, such as an ultrasonic radar.

[0053] In actual operation, Figure 2 As shown, 4 visual data acquisition devices and 12 ultrasonic data acquisition devices can be installed on the target vehicle; the 4 visual data acquisition devices can be installed in the middle of the front and rear of the vehicle, and near the left and right rearview mirrors; the 12 ultrasonic data acquisition devices can include 8 ultrasonic UPA probes and 4 ultrasonic APA probes, the 8 ultrasonic UPA probes can be installed at the front and rear of the vehicle, 4 at the front and rear of the vehicle, and 4 ultrasonic APA probes can be installed on both sides of the front and rear of the vehicle, with 1 on each side of the front and rear of the vehicle. Here, the parameters of the visual data acquisition device can be a horizontal field of view of 196°, a vertical field of view of 112°, a focal length of 1.51mm, 2 million pixels, and a maximum image resolution of 1920×1080; the detection range of the ultrasonic UPA probe can be 0.2m~2.5m; the detection range of the ultrasonic APA probe can be 0.2m~4m.

[0054] Step 120: convert the initial obstacle position, the initial parking space position, and the initial obstacle point position into the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

[0055] Among them, the target vehicle can be provided with a positioning module, and the positioning module can periodically output the positioning data of the target vehicle, specifically including the coordinates of the rear axle center of the target vehicle in the world coordinate system, and the angle between the front direction of the vehicle and the X-axis of the world coordinate system.

[0056] The purpose of this step is to use the fusion of visual data and ultrasonic data to deal with obstacles later.

[0057] The coordinate conversion in this step can be performed by the positioning data of the target vehicle output by the positioning module; specifically, Figure 3 As shown, the above step 120 may further include the following steps 310 to 330:

[0058] Step 310: calibrate the internal and external parameters of the visual data acquisition device to obtain a calibration table of the visual data acquisition device.

[0059] Among them, the internal parameter calibration uses the visual data acquisition equipment to shoot the chessboard calibration plate images in different postures and positions, and uses the calibration method (for example: Zhang Zhengyou calibration method) to calculate the internal parameters. Then, the calibration cloth is laid on the ground around the vehicle, and the position of the calibration cloth to the center of the rear axle of the vehicle is measured. Then, the external parameters of the visual data acquisition equipment relative to the center of the rear axle of the vehicle are calculated, and the calibration table of the visual data acquisition equipment is generated by combining the internal and external parameters.

[0060] Step 320: According to the calibration table, the initial obstacle position and the initial parking space position are converted to the first obstacle position and the first parking space position in the target vehicle coordinate system.

[0061] Among them, the initial obstacle position mainly refers to the edge ground point position in the obstacle detection frame output by the visual model after processing the image obtained by the visual data acquisition device; Figure 4 As shown, the edge of the detection frame is grounded to point P b1 and P b2 (Right now Figure 4 The ground point of the 2D box in the figure is taken as the initial obstacle position. Therefore, for the initial obstacle position, this step obtains P by querying the calibration table. b1 and P b2 The first obstacle position (x b1 ,y b1 ) and (x b2 ,y b2 ).

[0062] The initial parking space position refers to the four corner point positions (x p1 ,y p1 )、(x p2 ,y p2 )、(x p3 ,y p3 )、(x p4 ,y p4 ); Therefore, for the initial parking space position, this step obtains the four corner point positions (x p1 ,y p1 )、(x p2 ,y p2 )、(x p3 ,y p3 )、(x p4 ,y p4 ) is the first parking space position (x) in the target vehicle coordinate system p1 ,y p1 )、(x p2 ,y p2 )、(x p3 ,y p3 )、(x p4 ,y p4 ).

[0063] The initial obstacle point position is the position of the obstacle point in the target vehicle coordinate system obtained by the ultrasonic data acquisition device (x u ,y u ), therefore, the initial obstacle point position does not need to be converted to the position in the target vehicle coordinate system through the calibration table.

[0064] Step 330: According to the relationship between the characteristic object of the target vehicle in the time-synchronized positioning data and the world coordinate system, the first obstacle position, the first parking space position, and the initial obstacle point position are converted to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

[0065] Among them, by comparing the obstacle detection frame output by the visual model after processing the image obtained by the visual data acquisition device with the timestamp of the positioning data output by the positioning module, the time-synchronized positioning data can be determined; the time-synchronized positioning data includes the coordinates (X lb ,Y lb ), the angle θ between the vehicle head direction and the X-axis of the world coordinate system lb .

[0066] The feature object refers to the rear axle center and the front direction of the target vehicle.

[0067] For the first obstacle position (x b1 ,y b1 ) and (x b2 ,y b2 ) is the middle obstacle position in the world coordinate system (X b1 ,Y b1 ) and (X b2 ,Y b2 ) is determined by the following formula (1):

[0068]

[0069] For the first parking position (x p1 ,y p1 )、(x p2 ,y p2 )、(x p3 ,y p3 )、(x p4 ,y p4 )The target parking space position in the world coordinate system (X p1 ,Y p1 )、(X p2 ,Y p2 )、(X p3 ,Y p3 )、(X p4 ,Y p4 ) is determined by the following formula (2):

[0070]

[0071] For the initial obstacle point position (x u ,y u) is the position of the middle obstacle point in the world coordinate system (X u ,Y u ) is determined by the following formula (3):

[0072]

[0073] Step 130: Match the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point.

[0074] Among them, since the obstacle contour obtained by visual data has a large error, the obstacle points included in the obstacle are preliminarily determined by matching ultrasonic data with visual data to prepare for the subsequent correction of the obstacle contour.

[0075] The matching in this step can be determined by the relationship between the minimum distance between the obstacle point and the preset threshold; specifically, Figure 5 As shown, the above step 130 may further include the following steps 510 and 520:

[0076] Step 510: Determine a first distance between each intermediate obstacle point and each intermediate obstacle.

[0077] Among them, the calculation of the first distance can be divided into two cases, such as Figure 6 As shown in the figure, the first type is that the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is located inside the line segment; the second type is that the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is located outside the line segment; specifically, as Figure 7 As shown, the above step 310 may further include the following steps 710 and 720:

[0078] Step 710: When the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is located within the line segment, the projection distance of the vector formed by connecting the middle obstacle point and any one of the two edge grounding points on the line segment is determined as the first distance.

[0079] Among them, for the first case, the intermediate obstacle point P u (X u ,Y u ) to the middle obstacle (the line segment P formed by connecting the two edge grounding points b1 P b2 , where P b1 (X b1 ,Y b1 ), P b2 (X b2 ,Y b2 ))The distance d between ubAs shown in the following formula (4):

[0080]

[0081] Among them, d up P b1 P u Vector in P b1 P b2 The projection distance on.

[0082] d up Determined by the following formula (5):

[0083]

[0084] Among them, d b P b1 P b2 Length.

[0085] Step 720: When the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is outside the line segment, determine the distance between the middle obstacle point and each of the two edge grounding points, and determine the minimum distance as the first distance.

[0086] For the second case, the intermediate obstacle point P u (X u ,Y u ) to the middle obstacle (the line segment P formed by connecting the two edge grounding points b1 P b2 , where P b1 (X b1 ,Y b1 ), P b2 (X b2 ,Y b2 ))The distance d between ub As shown in the following formula (6):

[0087]

[0088] Among them, d b1 From the intermediate obstacle point to P b1 The distance between b2 From the intermediate obstacle point to P b2 The distance between.

[0089] Step 520: determine whether the minimum first distance is less than a preset threshold value. When the minimum first distance is less than the preset threshold value, determine that the intermediate obstacle point corresponding to the minimum first distance belongs to an intermediate obstacle.

[0090] The preset threshold may be set based on experience or by consulting literature, and is not limited here; for example, the preset threshold is set to 0.5 m.

[0091] Step 140: Use the target parking space position to correct the middle obstacle point in the middle obstacle to obtain the target obstacle point in the middle obstacle.

[0092] Among them, this step is mainly to remove the intermediate obstacle points with large errors to obtain more accurate target obstacle points. Figure 8 As shown, the above step 140 may further include the following steps 810 and 820:

[0093] Step 810: Determine the projection distance of the vector formed by connecting each corner point of the third preset side and each intermediate obstacle point in each target parking space on the third preset side to obtain the second distance and the third distance.

[0094] The third preset side is perpendicular to the second preset side.

[0095] like Fig. 9 As shown, the third preset side is the 1-2 side in the target parking space, the first preset side is the 2-3 side in the target parking space, the fourth preset side is the 3-4 side in the target parking space, and the second preset side is the 4-1 side in the target parking space; the column initialization obstacle point is the above-mentioned middle obstacle point.

[0096] The four corner points (X ip1 ,Y ip1 )、(X ip2 ,Y ip2 )、(X ip3 ,Y ip3 )、(X ip4 ,Y ip4 ), i is the parking space index, i = 1, 2…W, W is the number of target parking spaces; first, calculate the unit direction vector (t x12 ,t y12 ) is shown in the following formula (7):

[0097]

[0098] Secondly, calculate the projection distance of the vector from corner point 1 to each intermediate obstacle point on the third preset edge, that is, the second distance D k1 As shown in the following formula (8):

[0099] D k1 =(X ku -X ip1 )t x12 +(Y ku -Y ip1 )ty12 (8)

[0100] Calculate the projection distance of the vector from corner point 2 to each intermediate obstacle point on the third preset edge, that is, the third distance D k2 As shown in the following formula (9):

[0101] D k2 =(X ku -X ip2 )t x12 +(Y ku -Y ip2 )t y12 (9)

[0102] Among them, (X ku ,Y ku ) is the coordinate of the kth intermediate obstacle point (k=1,2…M, M is the number of intermediate obstacle points).

[0103] Step 820: If the product of the second distance and the third distance is a positive number, it means that the intermediate obstacle point is outside the target parking space; if the product of the second distance and the third distance is a negative number, it means that the intermediate obstacle point is inside the target parking space; remove the intermediate obstacle point inside the target parking space to obtain the target obstacle point in the intermediate obstacle.

[0104] The product of the second distance and the third distance is a positive number, that is, the second distance and the third distance are both positive numbers or both negative numbers; in this case, it indicates that the intermediate obstacle point is located outside the target parking space.

[0105] The product of the second distance and the third distance is a negative number, that is, one of the second distance and the third distance is a positive number and the other is a negative number; at this time, it indicates that the intermediate obstacle point is located in the target parking space.

[0106] The middle obstacle point located in the parking space is the middle obstacle point with a larger error and should be removed; thus, the above-mentioned target obstacle point is obtained.

[0107] Step 150: Determine the adjacent target parking spaces of the intermediate obstacle according to the distances between the target obstacle point and the first preset side and the second preset side in each target parking space.

[0108] The target parking space is composed of a third preset side, a first preset side, a fourth preset side, and a second preset side connected in sequence; the first preset side is parallel to the second preset side.

[0109] The adjacent target parking space of the intermediate obstacle can be determined from the target parking space according to the absolute minimum distance; specifically, Fig.10 As shown, the above step 150 may further include the following steps 1010 to 1030:

[0110] Step 1010: Determine the absolute minimum second distance and the absolute minimum third distance. If the absolute minimum second distance is a positive number, the target parking space is assigned to the first group; otherwise, the target parking space is assigned to the second group.

[0111] Among them, the absolute minimum second distance is the absolute minimum second distance between the target obstacle point and the second preset edge in the target parking space; the absolute minimum third distance is the absolute minimum third distance between the target obstacle point and the first preset edge in the target parking space.

[0112] The second distance D calculated in step 810 is k1 The absolute value of can represent the distance from the target obstacle point to the second preset edge in the target parking space; the third distance D calculated in step 820 above k2 The absolute value of can represent the distance from the target obstacle point to the first preset edge in the target parking space.

[0113] For the target parking space with index i, the distance from the target obstacle point to the second preset edge of the target parking space (i.e., the second distance) is counted, and the second distance with the smallest absolute value is recorded as S i14 If S i14 >0, the middle obstacle is on the left side of the second preset side of the target parking space; otherwise, the middle obstacle is on the right side of the second preset side of the target parking space.

[0114] For the target parking space with index i, the distance from the target obstacle point to the first preset edge of the target parking space (i.e., the third distance) is counted, and the third distance with the smallest absolute value is recorded as S i23 If S i23 >0, the middle obstacle is on the left side of the first preset side of the target parking space; otherwise, the middle obstacle is on the right side of the first preset side of the target parking space.

[0115] In actual operation, the first group may be recorded as group P, and the second group may be recorded as group N. If the second distance from the intermediate obstacle to the second preset side of the target parking space is a positive number, the target parking space is classified into group P (the first group); otherwise, the target parking space is classified into group N (the second group).

[0116] Step 1020: If the first group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the first group are both positive numbers, and the absolute minimum third distance is smaller than the absolute minimum second distance, the target parking space corresponding to the absolute minimum third distance is determined as the adjacent target parking space of the intermediate obstacle.

[0117] Among them, Fig.11 As shown, if group P (the first group) is not empty, S i14 With S i23are all positive numbers, and the middle obstacle is closer to the first preset edge of the target parking space (that is, the absolute value of the minimum third distance is smaller than the absolute value of the minimum second distance), then S i23 The corresponding target parking space is determined as the adjacent target parking space of the middle obstacle; the pillar obstacle point is the above-mentioned middle obstacle point.

[0118] Step 1030: If the second group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the second group are both negative numbers, and the absolute minimum second distance is less than the absolute minimum third distance, the target parking space corresponding to the absolute minimum second distance is determined as the adjacent target parking space of the intermediate obstacle.

[0119] Among them, if group N (the second group) is not empty, S i14 With S i23 If both are negative and the middle obstacle is closer to the second preset edge of the target parking space (i.e., the absolute minimum second distance is smaller than the absolute minimum third distance), then S i14 The corresponding target parking space is determined as the adjacent target parking space of the intermediate obstacle.

[0120] Step 160: Based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, and according to the side position and direction of the adjacent target parking space, determine the target projection point and the target contour length; and construct the target obstacle according to the target projection point and the target contour length.

[0121] Among them, the relative position relationship between the intermediate obstacle and the adjacent target parking space can be divided into three situations, such as Fig.12 As shown, the first type is the situation where there are adjacent target parking spaces on both sides of the middle obstacle; the second type is the situation where there is an adjacent target parking space on the left side of the middle obstacle; the third type is the situation where there is an adjacent target parking space on the right side of the middle obstacle; the pillar obstacle point is the above-mentioned middle obstacle point.

[0122] For the first case above, specifically, Fig.13 As shown, the above step 160 may further include the following steps 1310 to 1350:

[0123] Step 1310: When there are adjacent target parking spaces on both sides of the middle obstacle, determine the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on one side on the second preset side to obtain a fourth distance.

[0124] Among them, the attitude directions of two adjacent target parking spaces are usually consistent, such as Fig.11 As shown, the left adjacent target parking space can be selected, and the second preset side of the left adjacent target parking space is close to the middle obstacle. If the coordinates of the four corner points of the left adjacent target parking space are (X lp1 ,Y lp1 )、(Xlp2 ,Y lp2 )、(X lp3 ,Y lp3 )、(X lp4 ,Y lp4 ), then calculate the unit direction vector (t lx14 ,t ly14 ) is shown in the following formula (10):

[0125]

[0126] Determine the projection distance (i.e., the fourth distance) G of the vector from the target obstacle point to the corner point No. 1 in the left adjacent target parking space on the second preset edge in the left adjacent target parking space lk14 As shown in the following formula (11):

[0127] G lk14 =(X lp1 -X ku )t lx14 +(Y lp1 -Y ku )t ly14 (11)

[0128] Step 1320: determine a target obstacle point corresponding to the maximum fourth distance, and use a projection point of the target obstacle point corresponding to the maximum fourth distance on the second preset edge as a first target projection point.

[0129] Among them, the target obstacle point corresponding to the maximum fourth distance is recorded as P * (X * ,Y * ), the target obstacle point is located on the contour of the middle obstacle, and the projection point of the target obstacle point on the second preset edge in the left adjacent target parking space is determined as contour point No. 2. The coordinates of contour point No. 2 (X2, Y2) are determined by the following formula (12):

[0130]

[0131] Step 1330: Take the projection point of the target obstacle point corresponding to the maximum fourth distance on the first preset side in the adjacent target parking space on the other side as the second target projection point.

[0132] The projection point of the target obstacle point on the first preset edge in the right adjacent target parking space is determined as contour point No. 1. The coordinates (X1, Y1) of contour point No. 1 are determined by the following formula (13):

[0133]

[0134] Step 1340: taking the distance between the first target projection point and the second target projection point as the target contour length.

[0135] The distance between contour point 1 and contour point 2 is calculated as the target contour length. The target contour length L is determined by the following formula (14):

[0136]

[0137] Step 1350: determine the target obstacle formed by the target contour according to the target contour length, the first target projection point, the second target projection point, the second preset side direction in the adjacent target parking space on one side, and the first preset side direction in the adjacent target parking space on the other side.

[0138] Among them, starting from contour point 1, the contour point 4 is obtained by extending along the second preset side direction of the left adjacent target parking space according to the target contour length L; the coordinates of contour point 4 (X4, Y4) are determined by the following formula (15):

[0139]

[0140] Starting from contour point 2, the contour point 3 is obtained by extending along the second preset side direction of the left adjacent target parking space according to the target contour length L. The coordinates (X3, Y3) of contour point 3 are determined by the following formula (16):

[0141]

[0142] The target obstacle can be determined based on contour point No. 1, contour point No. 2, contour point No. 3, and contour point No. 4.

[0143] For the second case above, specifically, Fig.14 As shown, the above step 160 may further include the following steps 1410 to 1440:

[0144] Step 1410: When there is an adjacent target parking space on one side of the intermediate obstacle, determine the projection distance of a vector formed by connecting each target obstacle point with a corner point of a second preset side in the adjacent target parking space on the second preset side to obtain a fifth distance.

[0145] The implementation process of this step is the same as the implementation process of the above step 610, and will not be repeated here.

[0146] Step 1420: determine the target obstacle point corresponding to the maximum fifth distance, and use the projection point of the target obstacle point corresponding to the maximum fifth distance on the second preset edge as the third target projection point.

[0147] The implementation process of this step is the same as the implementation process of the above step 620, and will not be repeated here.

[0148] Step 1430: determine the projection distance of the vector formed by connecting each target obstacle point and the third target projection point on the third preset side, obtain the sixth distance, and use the sixth distance with the maximum absolute value as the target contour length.

[0149] First, determine the unit direction vector (t lx12 ,t ly12 ) is shown in the following formula (17):

[0150]

[0151] Secondly, determine the projection distance G of the vector from the target obstacle point to the third target projection point (i.e., contour point No. 2) on the third preset edge. lk12 As shown in the following formula (18):

[0152] G lk12 =(X2-X ku )t lx12 +(Y2-Y ku )t ly12 (18)

[0153] The sixth distance having the largest absolute value is taken as the target contour length L.

[0154] Step 1440: Determine the target obstacle formed by the target contour according to the third target projection point, the target contour length, the third preset edge direction in the adjacent target parking space, and the second preset edge direction.

[0155] Among them, starting from the contour point No. 2, extending along the opposite direction of the third preset edge in the adjacent target parking space according to the target contour length L, the coordinates of the contour point No. 1 (X1, Y1) are obtained as shown in the following formula (19):

[0156]

[0157] Starting from contour point No. 2, the coordinates of contour point No. 3 (X3, Y3) are obtained by extending along the direction of the second preset edge in the adjacent target parking space according to the target contour length L, and the coordinates of contour point No. 4 (X4, Y4) are obtained by the above formula (15).

[0158] The target obstacle can be determined based on contour point No. 1, contour point No. 2, contour point No. 3, and contour point No. 4.

[0159] For the third case, its implementation method is similar to that of the second case. Specifically, it is symmetrical with the implementation method of the second case, and will not be repeated here.

[0160] In actual operation, relevant software can be deployed on the domain controller for processing data, including key point visual models for detecting parking spaces, pillar obstacles, and other obstacles, driver software for parsing ultrasonic data, and software for detecting the contours of pillar obstacles, so as to implement the above steps 110 to 160.

[0161] The obstacle detection method integrating vision and ultrasound provided in the embodiment of the present application first obtains the initial obstacle position and the initial parking space position through a visual data acquisition device; obtains the initial obstacle point position through an ultrasonic data acquisition device; secondly, converts the initial obstacle position, the initial parking space position, and the initial obstacle point position into a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; thirdly, matches the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; and then, uses the target parking space position to match the intermediate obstacle in the intermediate obstacle. The intermediate obstacle point is corrected to obtain the target obstacle point in the middle obstacle; then, the adjacent target parking space of the middle obstacle is determined according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space; the target parking space is composed of the third preset side, the first preset side, the fourth preset side, and the second preset side connected in sequence; the first preset side is parallel to the second preset side; finally, based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, the target projection point and the target contour length are determined according to the side position and direction of the adjacent target parking space; the target obstacle is constructed according to the target projection point and the target contour length. In this way, by obtaining the obstacle position, fusing the obstacle point position, matching the obstacle point position with the obstacle position, and using the position and direction of the edge line of the adjacent parking space of the obstacle, the obstacle contour (i.e., the target obstacle) is finally generated, which effectively overcomes the defect of unstable detection results of column obstacles by visual methods, and corrects the column obstacle contour based on the parking space position information, improves the accuracy of ultrasonic method in detecting the column obstacle contour position, and thus provides strong perception data support for safe parking in parking spaces adjacent to column obstacles, which has great engineering application significance.

[0162] After introducing the obstacle detection method integrating vision and ultrasound according to the exemplary embodiment of the present disclosure, next, reference is made to Fig.15 An obstacle detection device 1500 integrating vision and ultrasonic waves according to an exemplary embodiment of the present disclosure will be described.

[0163] refer to Fig.15The obstacle detection device 1500 integrating vision and ultrasound comprises: an acquisition module 1510, configured to acquire an initial obstacle position and an initial parking space position through a visual data acquisition device; obstacles corresponding to the initial obstacle position include pillar obstacles; an initial obstacle point position is acquired through an ultrasonic data acquisition device; a coordinate conversion module 1520, configured to convert the initial obstacle position, the initial parking space position, and the initial obstacle point position into a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; a matching module 1530, configured to match the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; a correction module 1540, configured to The target parking space position is used to correct the middle obstacle point in the middle obstacle to obtain the target obstacle point in the middle obstacle; the first determination module 1550 is configured to determine the adjacent target parking space of the middle obstacle according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space; the target parking space is composed of a third preset side, a first preset side, a fourth preset side, and a second preset side connected in sequence; the first preset side is parallel to the second preset side; the second determination module 1560 is configured to determine the target projection point and the target contour length based on the relative position relationship between the middle obstacle and the adjacent target parking space according to the side position and direction of the adjacent target parking space; and the target obstacle is constructed according to the target projection point and the target contour length.

[0164] In one embodiment, the coordinate conversion module 1520 is configured to: calibrate the internal and external parameters of the visual data acquisition device to obtain a calibration table of the visual data acquisition device; according to the calibration table, convert the initial obstacle position and the initial parking space position to the first obstacle position and the first parking space position in the target vehicle coordinate system; according to the relationship between the characteristic object of the target vehicle in the time-synchronized positioning data and the world coordinate system, convert the first obstacle position, the first parking space position, and the initial obstacle point position to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

[0165] In one embodiment, the matching module 1530 is configured to: determine the first distance between each intermediate obstacle point and each intermediate obstacle; determine whether the minimum first distance is less than a preset threshold, and when the minimum first distance is less than the preset threshold, determine that the intermediate obstacle point corresponding to the minimum first distance belongs to an intermediate obstacle.

[0166] In one embodiment, the matching module 1530 is configured as follows: when the projection point of the intermediate obstacle point on the line segment formed by connecting the two edge grounding points of the intermediate obstacle is located within the line segment, the first distance is determined based on the projection distance of the vector formed by connecting the intermediate obstacle point and any one of the two edge grounding points on the line segment; when the projection point of the intermediate obstacle point on the line segment formed by connecting the two edge grounding points of the intermediate obstacle is located outside the line segment, the distance between the intermediate obstacle point and each of the two edge grounding points is determined, and the minimum distance is determined as the first distance.

[0167] In one embodiment, the correction module 1540 is configured to: determine the projection distance of the vector formed by connecting each corner point of the third preset side in each target parking space and each intermediate obstacle point on the third preset side to obtain the second distance and the third distance; the third preset side is perpendicular to the second preset side; if the product of the second distance and the third distance is a positive number, it indicates that the intermediate obstacle point is located outside the target parking space; if the product of the second distance and the third distance is a negative number, it indicates that the intermediate obstacle point is located inside the target parking space; remove the intermediate obstacle points located inside the target parking space to obtain the target obstacle point in the intermediate obstacle.

[0168] In one embodiment, the first determination module 1550 is configured to: determine the absolute minimum second distance and the absolute minimum third distance, and if the absolute minimum second distance is a positive number, the target parking space is divided into the first group; otherwise, the target parking space is divided into the second group; the absolute minimum second distance is the second distance with the smallest absolute value between the target obstacle point and the second preset side in the target parking space; the absolute minimum third distance is the third distance with the smallest absolute value between the target obstacle point and the first preset side in the target parking space; if the first group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the first group are both positive numbers, and the absolute minimum third distance is less than the absolute minimum second distance, then the target parking space corresponding to the absolute minimum third distance is determined as the adjacent target parking space of the middle obstacle; if the second group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the second group are both negative numbers, and the absolute minimum second distance is less than the absolute minimum third distance, then the target parking space corresponding to the absolute minimum second distance is determined as the adjacent target parking space of the middle obstacle.

[0169] In one embodiment, the second determination module 1560 is configured as follows: when there are adjacent target parking spaces on both sides of the middle obstacle, determine the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on one side on the second preset side to obtain a fourth distance; determine the target obstacle point corresponding to the maximum fourth distance, and use the projection point of the target obstacle point corresponding to the maximum fourth distance on the second preset side as the first target projection point; use the projection point of the target obstacle point corresponding to the maximum fourth distance on the first preset side in the adjacent target parking space on the other side as the second target projection point; use the distance between the first target projection point and the second target projection point as the target contour length; and determine the target obstacle formed by the target contour according to the target contour length, the first target projection point, the second target projection point, the direction of the second preset side in the adjacent target parking space on one side, and the direction of the first preset side in the adjacent target parking space on the other side.

[0170] In one embodiment, the second determination module 1560 is configured as follows: when there is an adjacent target parking space on one side of the intermediate obstacle, determine the projection distance of the vector formed by connecting each target obstacle point with the corner point of the second preset side in the adjacent target parking space on the second preset side to obtain a fifth distance; determine the target obstacle point corresponding to the maximum fifth distance, and use the projection point of the target obstacle point corresponding to the maximum fifth distance on the second preset side as the third target projection point; determine the projection distance of the vector formed by connecting each target obstacle point with the third target projection point on the third preset side to obtain a sixth distance, and use the sixth distance with the maximum absolute value as the target contour length; determine the target obstacle formed by the target contour according to the third target projection point, the target contour length, the third preset side direction, and the second preset side direction in the adjacent target parking space.

[0171] In one embodiment, the visual data acquisition device includes a camera, and the ultrasonic data acquisition device includes an ultrasonic radar; 4 visual data acquisition devices and 12 ultrasonic data acquisition devices are installed on the target vehicle, and the 4 visual data acquisition devices are installed in the middle of the front and rear of the vehicle, and at the left and right rearview mirror positions; the 12 ultrasonic data acquisition devices include 8 ultrasonic UPA probes and 4 ultrasonic APA probes, and 4 of the 8 ultrasonic UPA probes are installed at the front and rear of the vehicle, and 1 of the 4 ultrasonic APA probes is installed on both sides of the front and rear of the vehicle.

[0172] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0173] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0174] Fig.16 A schematic diagram of a computer device provided in an embodiment of the present application. The device can be integrated into a terminal device or a chip of a terminal device. The terminal can be a computing device with data processing capabilities.

[0175] The device includes: a processor 1601 , a storage medium 1602 and a bus 1603 .

[0176] The storage medium 1602 stores program instructions executable by the processor 1601. When the computer device 1600 is running, the processor 1601 communicates with the storage medium 1602 via the bus 1603. The processor 1601 executes the program instructions to perform the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0177] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute the above method embodiment when executed by a processor.

[0178] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0181] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0182] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An obstacle detection method integrating vision and ultrasound, characterized in that: include: Obtain the initial obstacle position and initial parking space position through visual data acquisition equipment; Obtain the initial obstacle point position through ultrasonic data acquisition equipment; The obstacles corresponding to the initial obstacle position include pillar obstacles; Convert the initial obstacle position, the initial parking space position, and the initial obstacle point position to a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; Matching the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; Using the target parking space position to correct the intermediate obstacle point in the intermediate obstacle to obtain a target obstacle point in the intermediate obstacle; Determine the adjacent target parking space of the intermediate obstacle according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space; the target parking space is composed of a third preset side, a first preset side, a fourth preset side, and a second preset side connected in sequence; the first preset side is parallel to the second preset side; Based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, and according to the edge position and direction of the adjacent target parking space, a target projection point and a target contour length are determined; and a target obstacle is constructed according to the target projection point and the target contour length.

2. The method according to claim 1, characterized in that: The step of converting the initial obstacle position, the initial parking space position, and the initial obstacle point position into a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position includes: Calibrate the internal and external parameters of the visual data acquisition device to obtain a calibration table of the visual data acquisition device; According to the calibration table, the initial obstacle position and the initial parking space position are converted to the first obstacle position and the first parking space position in the target vehicle coordinate system; According to the relationship between the characteristic object of the target vehicle in the time-synchronized positioning data and the world coordinate system, the first obstacle position, the first parking space position, and the initial obstacle point position are converted to the world coordinate system to obtain the intermediate obstacle position, the target parking space position, and the intermediate obstacle point position.

3. The method according to claim 1, characterized in that The matching of the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point includes: Determine a first distance between each of the intermediate obstacle points and each of the intermediate obstacles; It is determined whether the minimum first distance is less than a preset threshold value, and when the minimum first distance is less than the preset threshold value, it is determined that the intermediate obstacle point corresponding to the minimum first distance belongs to the intermediate obstacle.

4. The method according to claim 3, characterized in that The determining of a first distance between each of the intermediate obstacle points and each of the intermediate obstacles comprises: When the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is located within the line segment, the first distance is determined according to the projection distance on the line segment of the vector formed by connecting the middle obstacle point and any one of the two edge grounding points; When the projection point of the middle obstacle point on the line segment formed by connecting the two edge grounding points of the middle obstacle is outside the line segment, the distance between the middle obstacle point and each of the two edge grounding points is determined, and the minimum distance is determined as the first distance.

5. The method according to claim 1, characterized in that The step of using the target parking space position to correct the intermediate obstacle point in the intermediate obstacle to obtain the target obstacle point in the intermediate obstacle includes: Determine the projection distance of a vector formed by connecting each corner point of the third preset side in each of the target parking spaces with each of the intermediate obstacle points on the third preset side to obtain a second distance and a third distance; the third preset side is perpendicular to the second preset side; If the product of the second distance and the third distance is a positive number, it means that the intermediate obstacle point is located outside the target parking space; if the product of the second distance and the third distance is a negative number, it means that the intermediate obstacle point is located inside the target parking space; remove the intermediate obstacle point located inside the target parking space to obtain the target obstacle point in the intermediate obstacle.

6. The method according to claim 1, characterized in that The step of determining the adjacent target parking space of the intermediate obstacle according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space comprises: Determine the absolute minimum second distance and the absolute minimum third distance. If the absolute minimum second distance is a positive number, the target parking space is classified into the first group; otherwise, the target parking space is classified into the second group; the absolute minimum second distance is the absolute minimum second distance between the target obstacle point and the second preset side in the target parking space; the absolute minimum third distance is the absolute minimum third distance between the target obstacle point and the first preset side in the target parking space; If the first group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the first group are both positive numbers, and the absolute minimum third distance is smaller than the absolute minimum second distance, the target parking space corresponding to the absolute minimum third distance is determined as the adjacent target parking space of the intermediate obstacle; If the second group is not empty, and the absolute minimum second distance and the absolute minimum third distance in the second group are both negative numbers, and the absolute minimum second distance is less than the absolute minimum third distance, the target parking space corresponding to the absolute minimum second distance is determined as the adjacent target parking space of the intermediate obstacle.

7. The method according to claim 1, characterized in that Based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, the target projection point and the target contour length are determined according to the edge position and direction of the adjacent target parking space; Constructing a target obstacle according to the target projection point and the target contour length, comprising: When there are adjacent target parking spaces on both sides of the middle obstacle, determine the projection distance of a vector formed by connecting each target obstacle point with a corner point of the second preset side in the adjacent target parking space on one side on the second preset side to obtain a fourth distance; Determine the target obstacle point corresponding to the maximum fourth distance, and use the projection point of the target obstacle point corresponding to the maximum fourth distance on the second preset side as the first target projection point; Taking the projection point of the target obstacle point corresponding to the maximum fourth distance on the first preset side of the adjacent target parking space on the other side as the second target projection point; Taking the distance between the first target projection point and the second target projection point as the target contour length; The target obstacle formed by the target contour is determined according to the target contour length, the first target projection point, the second target projection point, the second preset edge direction in the adjacent target parking space on one side, and the first preset edge direction in the adjacent target parking space on the other side.

8. The method according to claim 1, characterized in that Based on the relative position relationship between the intermediate obstacle and the adjacent target parking space, the target projection point and the target contour length are determined according to the edge position and direction of the adjacent target parking space; Constructing a target obstacle according to the target projection point and the target contour length, including: When the adjacent target parking space exists on one side of the intermediate obstacle, determine the projection distance of a vector formed by connecting each target obstacle point with a corner point of a second preset side in the adjacent target parking space on the second preset side to obtain a fifth distance; Determine the target obstacle point corresponding to the maximum fifth distance, and use the projection point of the target obstacle point corresponding to the maximum fifth distance on the second preset side as the third target projection point; Determine the projection distance of the vector formed by connecting each of the target obstacle points and the third target projection point on the third preset side to obtain a sixth distance, and use the sixth distance with the largest absolute value as the target contour length; The target obstacle formed by the target contour is determined according to the third target projection point, the target contour length, the third preset edge direction and the second preset edge direction in the adjacent target parking space.

9. The method according to claim 1, characterized in that: The visual data acquisition device includes a camera, and the ultrasonic data acquisition device includes an ultrasonic radar; 4 of the visual data acquisition devices and 12 of the ultrasonic data acquisition devices are installed on the target vehicle, and the 4 visual data acquisition devices are installed in the middle position between the front and rear of the vehicle, and at the positions of the left and right rearview mirrors; the 12 ultrasonic data acquisition devices include 8 ultrasonic UPA probes and 4 ultrasonic APA probes, and 4 of the 8 ultrasonic UPA probes are installed at the front and rear of the vehicle, and 1 of the 4 ultrasonic APA probes is installed on both sides of the front and rear of the vehicle.

10. An obstacle detection device integrating vision and ultrasound, characterized in that: include: An acquisition module is configured to acquire an initial obstacle position and an initial parking space position through a visual data acquisition device; Obtain the initial obstacle point position through ultrasonic data acquisition equipment; The obstacles corresponding to the initial obstacle position include pillar obstacles; A coordinate conversion module is configured to convert the initial obstacle position, the initial parking space position, and the initial obstacle point position into a world coordinate system to obtain an intermediate obstacle position, a target parking space position, and an intermediate obstacle point position; A matching module is configured to match the intermediate obstacle position with the intermediate obstacle point position to obtain a corresponding relationship between the intermediate obstacle and the intermediate obstacle point; a correction module, configured to correct the intermediate obstacle point in the intermediate obstacle by using the target parking space position to obtain a target obstacle point in the intermediate obstacle; A first determination module is configured to determine the adjacent target parking space of the intermediate obstacle according to the distance between the target obstacle point and the first preset side and the second preset side in each target parking space; The second preset side is parallel to the first preset side; The second determination module is configured to determine the target projection point and the target contour length based on the relative position relationship between the intermediate obstacle and the adjacent target parking space and according to the edge position and direction of the adjacent target parking space; and construct the target obstacle according to the target projection point and the target contour length.

Images