A method and system for intelligent parking system obstacle information fusion

Abstract

The present invention discloses an obstacle information fusion method and system for an intelligent parking system, including: obtaining ultrasonic information detected by a vehicle-mounted ultrasonic sensor component, and obtaining obstacle information of space obstacles around the vehicle based on the ultrasonic information; obtaining image information captured by a vehicle-mounted camera component, and obtaining obstacle information of visual obstacles around the vehicle based on the image information; synchronizing obstacle information of space obstacles and visual obstacles around the vehicle in space and time; determining one or more target obstacle points based on the synchronized obstacle information of space obstacles and visual obstacles around the vehicle, and performing information fusion to output obstacle information of one or more target obstacle points in a preset format. Through the present invention, the obstacle information detected by the ultrasonic sensor component and the camera component can be fused, and the obstacle information can be output in a unified format, thereby improving the reliability and accuracy of the obstacle information.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic parking, and in particular to an obstacle information fusion method and system of an intelligent parking system. Background Art

[0002] With the rapid development of society, the intelligence of automobiles is constantly improving, and intelligent driving assistance systems are gradually being widely used. Among them, automatic parking technology is an important application of intelligent driving assistance systems in parking scenarios, and it will become more and more popular. The automatic parking system is a comprehensive system that integrates perception, decision-making, planning and control functions. It senses the surrounding environment of the vehicle through sensors, automatically searches for spatial parking spaces or ground parking space markings, such as parking space lines, etc., and then plans the automatic parking trajectory based on the search results, and automatically controls the vehicle to park at the target parking location.

[0003] Existing automatic parking systems can be mainly divided into two categories. One is a parking system that relies only on ultrasonic sensors. It uses a single ultrasonic sensor to complete functions such as parking space search and obstacle detection. The perception dimension is relatively single, and it can only identify obstacle parking spaces but not linear parking spaces. The other is a fusion parking system that integrates cameras and ultrasonic sensors. Through the fusion processing of ultrasonic information and camera information, it can realize the identification and parking of obstacle parking spaces and linear parking spaces.

[0004] Regarding the fusion parking system, its sensor fusion strategy is a key technology in the intelligent parking perception layer. The quality of the design performance directly affects the decision-making, planning and control of parking, and is also related to the safety of parking. The current mainstream automatic parking system sensor solution consists of 12 ultrasonic sensors and 4 panoramic cameras. Among them, the ultrasonic sensor mainly calculates the relative distance between the reflection point and the vehicle by sending and receiving ultrasonic waves and their reflected echoes, and then the controller calculates the relative position of the reflection point and the vehicle according to the triangulation positioning principle; due to the characteristics of the ultrasonic sensor, the ultrasonic detection distance is often short, the detection results are discontinuous, and the type and size of the obstacle cannot be accurately given. The camera collects visual images, and then the controller uses intelligent algorithms such as deep learning to identify specific objects or patterns in the image, which is often greatly affected by external environmental conditions such as light.

[0005] In addition, most current technical solutions have a low degree of integration, and only simply superimpose and classify ultrasonic information and camera information, or directly use them according to their functions. For example, ultrasonic sensors are used to realize spatial parking space search and obstacle detection, but fail to identify the type of obstacles; cameras are only used for linear parking space search and do not participate in obstacle detection.

[0006] In summary, the obstacle information fusion technology of the existing automatic parking system still needs to be improved. Summary of the invention

[0007] The purpose of the present invention is to propose an obstacle information fusion method and system for an intelligent parking system, so as to realize the fusion of obstacle information detected by an ultrasonic sensor component and a camera component, increase the dimension of obstacle information, and improve the reliability and accuracy of obstacle information.

[0008] To achieve the above-mentioned purpose, the first aspect of the present invention provides an obstacle information fusion method for an intelligent parking system, comprising:

[0009] Acquire ultrasonic information detected by the vehicle-mounted ultrasonic sensor component, and obtain obstacle information of obstacles in the space around the vehicle based on the ultrasonic information;

[0010] Obtaining image information captured by the vehicle-mounted camera assembly, and obtaining obstacle information of visual obstacles around the vehicle based on the image information;

[0011] Synchronize obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time;

[0012] One or more target obstacle points are determined according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and the obstacle information of the one or more target obstacle points in a preset format is output by performing information fusion.

[0013] Optionally, where:

[0014] If any spatial obstacle and any visual obstacle refer to the same target obstacle, information fusion is performed based on the obstacle information of the spatial obstacle and the visual obstacle to obtain obstacle information of a set of obstacle points in a preset format representing the target obstacle;

[0015] If any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is regarded as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle;

[0016] If any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained based on the obstacle information of the visual obstacle.

[0017] Optionally, obtaining obstacle information of obstacles in a space around the vehicle according to the ultrasonic information includes:

[0018] Calculating position information of multiple obstacle points relative to the center of the vehicle based on the ultrasonic information;

[0019] According to the position information of the plurality of obstacle points relative to the center of the vehicle, the plurality of obstacle points are grouped to obtain one or more groups of obstacle points, and obstacle information of obstacles in the space around the vehicle is further obtained;

[0020] Among them, multiple obstacle points in the same group represent multiple points on the same spatial obstacle; the obstacle information of each spatial obstacle includes a group number of a group of obstacle points and position information of all obstacle points in the group.

[0021] Optionally, obtaining obstacle information of visual obstacles around the vehicle according to the image information includes:

[0022] The image information is processed by image recognition using a pre-trained deep learning network model to output obstacle information of visual obstacles around the vehicle; wherein the obstacle information of the visual obstacle includes the location information, width, category, category confidence and timestamp of the visual obstacle; the timestamp indicates the time when the on-board camera component captures the visual obstacle.

[0023] Optionally, synchronizing the obstacle information of the spatial obstacles and visual obstacles around the vehicle in space and time includes:

[0024] Mapping the position information of all obstacle points and visual obstacles to the same vehicle coordinate system, obtaining the coordinates of all obstacle points and visual obstacles in the same vehicle coordinate system, so as to achieve spatial synchronization; wherein the vehicle coordinate system is a two-dimensional Cartesian coordinate system established with the center of the vehicle as the origin;

[0025] The compensation time of the visual obstacle is calculated according to the timestamp and the preset time compensation algorithm; the historical running status of the vehicle is obtained, and the coordinates of the visual obstacle in the vehicle coordinate system are updated according to the compensation time and the historical running status of the vehicle to achieve time synchronization.

[0026] Optionally, the method of determining whether the spatial obstacle and the visual obstacle refer to the same target obstacle is as follows:

[0027] In the vehicle coordinate system, taking the coordinates of any visual obstacle as the origin and 1 / 2 of the width of the visual obstacle as the radius, a circular area is established as the visual target to be matched area;

[0028] The ratio of the number of obstacle points in each group of obstacle points located in the visual target to be matched area to the total number of obstacle points in the group of obstacle points is calculated respectively, and it is determined whether the largest ratio is greater than a preset threshold; if so, the spatial obstacle corresponding to the largest ratio and the visual obstacle refer to the same target obstacle; if not, there is no spatial obstacle that refers to the same target obstacle as the visual obstacle.

[0029] Optionally, the obstacle information in the preset format includes coordinates, categories, and category confidence of the target obstacle;

[0030] Wherein, the obtaining obstacle information of a set of obstacle points in a preset format representing the target obstacle by fusing the obstacle information of the spatial obstacle and the visual obstacle comprises: taking the coordinates of the set of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system as the coordinates of the set of obstacle points representing the target obstacle; taking the category and category confidence of the visual obstacle as the category and category confidence of the set of obstacle points representing the target obstacle;

[0031] The step of obtaining obstacle information of a preset format of a group of obstacle points representing the target obstacle according to the obstacle information of the spatial obstacle includes: using coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system as coordinates of a group of obstacle points representing the target obstacle; using “spatial obstacle” as a category of the group of obstacle points representing the target obstacle, and setting the category confidence to 100%;

[0032] The obtaining, according to the obstacle information of the visual obstacle, obstacle information in a preset format representing an obstacle point of the target obstacle comprises: taking the coordinates of the visual obstacle in the vehicle coordinate system as the coordinates of an obstacle point representing the target obstacle; and taking the category and category confidence of the visual obstacle as the category and category confidence of an obstacle point representing the target obstacle.

[0033] The second aspect of the present invention provides an intelligent parking system obstacle information fusion system, comprising:

[0034] A spatial obstacle detection unit, used to obtain ultrasonic information detected by the vehicle-mounted ultrasonic sensor assembly, and obtain obstacle information of spatial obstacles around the vehicle based on the ultrasonic information;

[0035] A visual obstacle detection unit, used to obtain image information captured by the vehicle-mounted camera assembly, and obtain obstacle information of visual obstacles around the vehicle based on the image information;

[0036] a space-time synchronization unit for synchronizing obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time; and

[0037] The information fusion unit is used to determine one or more target obstacle points according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and perform information fusion to output the obstacle information of the one or more target obstacle points in a preset format.

[0038] Optionally, where:

[0039] When any spatial obstacle and any visual obstacle refer to the same target obstacle, information fusion is performed based on the obstacle information of the spatial obstacle and the visual obstacle to obtain obstacle information of a set of obstacle points in a preset format representing the target obstacle;

[0040] When any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is regarded as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle;

[0041] When any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained according to the obstacle information of the visual obstacle.

[0042] Optionally, the obstacle information in the preset format includes coordinates, categories, and category confidence of the target obstacle;

[0043] Wherein, when any spatial obstacle and any visual obstacle refer to the same target obstacle, the coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a group of obstacle points representing the target obstacle; the category and category confidence of the visual obstacle are used as the category and category confidence of a group of obstacle points representing the target obstacle;

[0044] Wherein, when there is no visual obstacle that refers to the same target obstacle as any spatial obstacle, the coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a group of obstacle points representing the target obstacle; "spatial obstacle" is used as the category of the group of obstacle points representing the target obstacle, and the category confidence is set to 100%;

[0045] Among them, when any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as it, the coordinates of the visual obstacle in the vehicle coordinate system are used as the coordinates of an obstacle point representing the target obstacle; and the category and category confidence of the visual obstacle are used as the category and category confidence of an obstacle point representing the target obstacle.

[0046] In summary, the embodiments of the present invention propose an obstacle information fusion method and system for an intelligent parking system, which fully utilizes the characteristics of an ultrasonic sensor component and a camera component, obtains obstacle information of spatial obstacles around a vehicle according to the ultrasonic information, and obtains obstacle information of visual obstacles around the vehicle according to the image information; synchronizes the obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time; determines one or more target obstacle points according to the synchronized obstacle information of spatial obstacles and visual obstacles around the vehicle, and performs information fusion, and finally outputs obstacle information of one or more target obstacle points in a preset format in a unified format; increases the dimension of obstacle information, improves the reliability and accuracy of obstacle information, and provides a solid perception foundation for subsequent links such as decision-making and control of automatic parking.

[0047] Other features and advantages of the present invention will be set forth in the description which follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 The figure is a flow chart of an obstacle information fusion method of an intelligent parking system in one embodiment of the present invention.

[0050] Figure 2 Schematic diagram of spatial obstacle detection in one embodiment of the present invention.

[0051] Figure 3 Schematic diagram of visual obstacle detection in one embodiment of the present invention.

[0052] Figure 4 The figure is a schematic diagram of the structure of an obstacle information fusion system of an intelligent parking system in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In addition, in order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, means well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0054] See also Figure 1 An embodiment of the present invention provides an obstacle information fusion method for an intelligent parking system, comprising:

[0055] Step S1, acquiring ultrasonic information detected by the vehicle-mounted ultrasonic sensor component, and obtaining obstacle information of obstacles in the space around the vehicle according to the ultrasonic information;

[0056] As an application example, the ultrasonic assembly includes 12 ultrasonic sensors disposed on the front and rear body of the vehicle;

[0057] Exemplarily, obtaining obstacle information of obstacles in the space around the vehicle according to the ultrasonic information includes:

[0058] Step S11, calculating the position information of multiple obstacle points relative to the center of the vehicle according to the ultrasonic information;

[0059] Specifically, the ultrasonic sensor sends ultrasonic signals in real time and receives corresponding echo signals. According to the echo signals, the position coordinates (MX′, MY′) of the reflection point (obstacle point) relative to the ultrasonic sensor can be calculated by the triangulation positioning principle. Then, according to the position of the ultrasonic sensor relative to the center of the vehicle, the coordinate values ​​of the obstacle point obtained by the 12 probes can be transformed to obtain the position coordinates (MX, MY) of the obstacle point based on the center of the vehicle (0, 0);

[0060] Step S12: according to the position information of the plurality of obstacle points relative to the center of the vehicle, that is, the position coordinates (MX, MY) of the obstacle points obtained in step S11, the plurality of obstacle points are grouped to obtain one or more groups of obstacle points, and obstacle information of obstacles in the space around the vehicle is further obtained;

[0061] Wherein, multiple obstacle points in the same group represent multiple points on the same spatial obstacle; the obstacle information of each spatial obstacle includes a group number of a group of obstacle points and the position information of all obstacle points in the group;

[0062] Specifically, for the same obstacle, multiple ultrasonic reflection points, i.e., obstacle points, may be obtained; in this embodiment, for ease of description, the ultrasonic reflection points are described as obstacle points; therefore, all obstacle points need to be grouped to determine how many spatial obstacles exist in the vehicle's surrounding environment. In step S2, a group of obstacle points corresponds to one spatial obstacle;

[0063] Specifically, grouping all obstacle points is specifically grouping according to the positional relationship between all obstacle points. For example, obstacle points with adjacent coordinates may be grouped into the same group.

[0064] In one application scenario, for example Figure 2 As shown in the figure, both the vehicle and the pedestrian were detected by the ultrasonic sensor, and there were two sets of valid spatial obstacle data respectively. However, the ice cream cone was small and far away from the vehicle, and the number of side probes was small, so the number of echoes obtained was insufficient, and the reflection point of the ice cream cone could not be obtained;

[0065] Step S2, acquiring image information captured by the vehicle-mounted camera assembly, and obtaining obstacle information of visual obstacles around the vehicle based on the image information;

[0066] As an application example, the camera assembly includes four cameras arranged in the front, rear, left and right directions of the vehicle to achieve 360-degree coverage of the vehicle's surrounding environment without blind spots. At the same sampling moment, four images can be captured; that is, at the same sampling moment, the image information includes four images;

[0067] For example, see Figure 3 , the obtaining obstacle information of visual obstacles around the vehicle according to the image information includes:

[0068] Performing image recognition processing on the image information using a pre-trained deep learning network model, and outputting obstacle information of visual obstacles around the vehicle; wherein the obstacle information of the visual obstacles includes the location information, width, category, category confidence and timestamp of the visual obstacles; the timestamp indicates the time when the on-board camera component captures the visual obstacle, that is, the shooting time corresponding to the image;

[0069] Specifically, through deep learning, many different types of obstacles can be identified. For example, typical obstacle types include: pedestrians, parking limiters, ice cream cones, no-parking facilities, etc.

[0070] Specifically, the information output by the deep learning network model in this embodiment includes: visual obstacle category, visual obstacle location information, visual obstacle width, category confidence; the visual obstacle location information is a location point; Figure 3 For ease of understanding, the visual obstacles in Figure 3 A horizontal line is added to the position of the visual obstacle in the image. It can be understood that the horizontal line is the part where the visual obstacle contacts the ground. The width of the horizontal line is the width of the visual obstacle, and the center point of the horizontal line is the position point of the visual obstacle. It should be noted that in the actual processing process, there is no recognition or generation of horizontal lines. The deep learning network model directly recognizes the visual obstacles and outputs the corresponding information.

[0071] Step S3, synchronizing obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time;

[0072] Specifically, due to the different sampling times of the camera and ultrasonic sensor, as well as the delay in signal transmission, the targets perceived by different sensors will have spatial errors due to the time difference. If time synchronization is not performed to compensate, the decision control module of the parking system will not accurately locate the target, resulting in decision deviation and affecting the parking effect.

[0073] Exemplarily, synchronizing the obstacle information of the spatial obstacles and visual obstacles around the vehicle in space and time includes:

[0074] Step S31, mapping the position information of all obstacle points and visual obstacles to the same vehicle coordinate system, obtaining the coordinates of all obstacle points and visual obstacles in the same vehicle coordinate system, so as to achieve spatial synchronization; wherein the vehicle coordinate system is a two-dimensional Cartesian coordinate system established with the center of the vehicle as the origin;

[0075] Specifically, since the sampling time of the camera and the ultrasonic sensor is different, the coordinates of all obstacle points and the coordinates of the visual obstacles obtained in step S31 are not collected at the same time, so time synchronization of step S32 is also required;

[0076] Step S32, calculating the compensation time of the visual obstacle according to the timestamp and the preset time compensation algorithm; obtaining the historical running status of the vehicle, and updating the coordinates of the midpoint of the visual obstacle in the vehicle coordinate system according to the compensation time and the historical running status of the vehicle to achieve time synchronization;

[0077] It should be noted that the purpose of the compensation time is to compensate for the time asynchrony in detecting spatial and visual obstacles caused by the sampling time difference between the camera and the ultrasonic sensor. The specific situation of this asynchrony is related to the technical parameters of the camera and the ultrasonic sensor. Therefore, the preset time compensation algorithm needs to be set according to the sampling time difference between the camera and the ultrasonic sensor in actual application;

[0078] Due to the time asynchrony, after the compensation time synchronizes the time, if the vehicle is in motion during the time period corresponding to the compensation time, the position of the obstacle should change relative to the position before the compensation time. It can be understood that, in combination with the vehicle kinematic model and the running state of the vehicle during this period, including the vehicle speed and heading angle, it can be further determined how to update the coordinates of the midpoint of the visual obstacle in the vehicle coordinate system, so as to achieve the synchronization of the coordinates of the obstacle point and the coordinates of the visual obstacle in time.

[0079] The above is only an example of spatial and temporal synchronization of spatial obstacles and visual obstacles detected by different sensor components in this embodiment. Of course, it can also be implemented in combination with other spatial and temporal synchronization methods, which are all within the protection scope of the embodiments of the present invention.

[0080] Step S4, determining one or more target obstacle points according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and performing information fusion to output obstacle information of the one or more target obstacle points in a preset format;

[0081] Specifically, as shown in Table 1 below, when performing information fusion, the method of this embodiment proposes three corresponding information fusion methods for three different target obstacle detection results;

[0082] Table 1 - Information Fusion

[0083]

[0084]

[0085] Exemplarily, the obstacle information in the preset format includes the coordinates of the target obstacle point, the category of the target obstacle point, and the confidence of the category of the obstacle point; the category of the target obstacle point is, for example, "ice cream cone", "vehicle", "bicycle", "pedestrian", "space obstacle", etc.; the confidence of the category of the obstacle point is expressed in percentage;

[0086] in:

[0087] If any spatial obstacle and any visual obstacle refer to the same target obstacle, obstacle information of a preset format representing a group of obstacle points of the target obstacle is obtained by performing information fusion based on the obstacle information of the spatial obstacle and the visual obstacle; wherein the coordinates of the group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of the group of obstacle points representing the target obstacle; and the category and category confidence of the visual obstacle are used as the category and category confidence of the group of obstacle points representing the target obstacle;

[0088] Specifically, assuming that there are 6 obstacle points in a set of obstacle points corresponding to the spatial obstacle, the obstacle information representing the target obstacle that is finally output includes the coordinates, categories, and category confidences corresponding to the 6 obstacle points;

[0089] If any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is taken as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle; wherein the coordinates of the set of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are taken as the coordinates of the set of obstacle points representing the target obstacle; "spatial obstacle" is taken as the category of the set of obstacle points representing the target obstacle, and the category confidence is set to 100%;

[0090] Specifically, assuming that there are 8 obstacle points in a set of obstacle points corresponding to the spatial obstacle, the obstacle information representing the target obstacle that is finally output includes the coordinates corresponding to the 8 obstacle points, the category is "spatial obstacle" and the category confidence is 100%; if any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as it, then the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained based on the obstacle information of the visual obstacle;

[0091] If any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is taken as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained according to the obstacle information of the visual obstacle; wherein the coordinates of the visual obstacle in the vehicle coordinate system are taken as the coordinates of an obstacle point representing the target obstacle; and the category and category confidence of the visual obstacle are taken as the category and category confidence of an obstacle point representing the target obstacle;

[0092] Specifically, since a visual obstacle corresponds to only one position point, the obstacle information representing the target obstacle that is finally output includes the coordinates, category, and category confidence corresponding to one obstacle point.

[0093] Exemplarily, the method of determining whether the spatial obstacle and the visual obstacle refer to the same target obstacle is as follows:

[0094] In the vehicle coordinate system, taking the coordinates of the position point of any visual obstacle as the origin and 1 / 2 of the width of the visual obstacle as the radius, a circular area is established as the visual target to be matched area;

[0095] The ratio of the number of obstacle points in each group of obstacle points located in the visual target to be matched area to the total number of obstacle points in the group of obstacle points is calculated respectively, and it is determined whether the largest ratio is greater than a preset threshold; if so, the spatial obstacle corresponding to the largest ratio and the visual obstacle refer to the same target obstacle; if not, there is no spatial obstacle that refers to the same target obstacle as the visual obstacle.

[0096] In summary, in the embodiment of the present invention, firstly, a multi-channel ultrasonic sensor and a multi-channel panoramic camera are used to collect ultrasonic echo information and visual image information respectively, and then the controller performs original algorithm processing on the data of the two sensors respectively to obtain spatial obstacle target information and visual obstacle target information respectively. After spatial and temporal synchronization, the visual obstacle information on the camera side and the spatial obstacle information on the ultrasonic sensor side are subjected to target matching and information fusion processing to generate unified fused obstacle information. Then, the decision, planning and control modules use the fused obstacle information as a reference to complete functions such as parking space fusion, path planning and path tracking.

[0097] The following describes an example of the application of the obstacle information of the target obstacle output by the method of this embodiment;

[0098] When performing parking space fusion, if there is a target obstacle point in the parking space to be released, its category will be determined. If the category is "spatial obstacle", the parking space will be cancelled; if the category is "ice cream cone", "vehicle", "bicycle" and other typical objects that are prohibited from parking, and the corresponding category confidence exceeds a certain set threshold, the parking space will be cancelled, otherwise it will still be released as a valid parking space; if the category is "pedestrian" or other movable objects, it will be released as a suspicious parking space, and the user will decide whether to park in the parking space;

[0099] When tracking the parking path, the obstacle avoidance strategy needs to be determined based on the obstacle information of the target obstacle point. If there is a target obstacle point on the vehicle's driving trajectory, its category is determined. If the obstacle category is "spatial obstacle", the vehicle is braked as soon as possible to prevent a collision. If the obstacle category is a target with a higher safety level such as "person" or "vehicle", the vehicle can be braked immediately as long as the confidence reaches a lower set threshold. If the obstacle category is "limit rod", the target parking position should be adjusted according to its relative distance to avoid the rear wheel of the vehicle hitting the limit device.

[0100] It should be noted that the above is only an example of the application of the target obstacle information output by the method of the embodiment, and it should be understood that the application of the target obstacle information of this embodiment is not limited to the above method.

[0101] The embodiments of the present invention have at least the following advantages:

[0102] (1) This embodiment method proposes a data-level fusion method for obstacle information fusion, and provides corresponding data fusion methods for all situations of perception target matching, which is highly practical.

[0103] (2) The fused obstacle information obtained by the method of this embodiment has both the relatively stable and accurate characteristics of ultrasonic information and the advantages of more dimensions of visual information. It can be used in multiple links of automatic parking decision-making and control and has a very wide range of applicability.

[0104] (3) The fused obstacle information obtained by the method of this embodiment is highly compatible. At the level of the automatic parking software, the original two types of perception information are fused into one type of information with a concise format. The decision-making and control modules can quickly use the fused obstacle information without making major changes.

[0105] (4) The method of this embodiment is developed based on the existing hardware solution, without increasing additional hardware costs.

[0106] See also Figure 4 Another embodiment of the present invention further provides an intelligent parking system obstacle information fusion system, which can be used to implement the above embodiment method. The system of this embodiment includes:

[0107] The spatial obstacle detection unit 1 is used to obtain ultrasonic information detected by the vehicle-mounted ultrasonic sensor component, and obtain obstacle information of spatial obstacles around the vehicle based on the ultrasonic information;

[0108] The visual obstacle detection unit 2 is used to obtain image information captured by the vehicle-mounted camera assembly, and obtain obstacle information of visual obstacles around the vehicle based on the image information;

[0109] A space-time synchronization unit 3, used for synchronizing obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time; and

[0110] The information fusion unit 4 is used to determine one or more target obstacle points according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and perform information fusion to output the obstacle information of the one or more target obstacle points in a preset format.

[0111] Exemplarily, wherein:

[0112] When any spatial obstacle and any visual obstacle refer to the same target obstacle, information fusion is performed based on the obstacle information of the spatial obstacle and the visual obstacle to obtain obstacle information of a set of obstacle points in a preset format representing the target obstacle;

[0113] When any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is regarded as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle;

[0114] When any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained according to the obstacle information of the visual obstacle.

[0115] Exemplarily, the obstacle information in the preset format includes the coordinates, category, and category confidence of the target obstacle;

[0116] Wherein, when any spatial obstacle and any visual obstacle refer to the same target obstacle, the coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a group of obstacle points representing the target obstacle; the category and category confidence of the visual obstacle are used as the category and category confidence of a group of obstacle points representing the target obstacle;

[0117] Wherein, when there is no visual obstacle that refers to the same target obstacle as any spatial obstacle, the coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a group of obstacle points representing the target obstacle; "spatial obstacle" is used as the category of the group of obstacle points representing the target obstacle, and the category confidence is set to 100%;

[0118] Among them, when any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as it, the coordinates of the visual obstacle in the vehicle coordinate system are used as the coordinates of an obstacle point representing the target obstacle; and the category and category confidence of the visual obstacle are used as the category and category confidence of an obstacle point representing the target obstacle.

[0119] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] It should be noted that the system described in the above embodiment corresponds to the method described in the above embodiment. Therefore, the undetailed parts of the system described in the above embodiment can be obtained by referring to the contents of the method described in the above embodiment, and will not be repeated here.

[0121] Furthermore, if the obstacle information fusion system of the intelligent parking system described in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0122] Specifically, the computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0123] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An obstacle information fusion method for an intelligent parking system, characterized in that: include: Acquire ultrasonic information detected by the vehicle-mounted ultrasonic sensor component, and obtain obstacle information of obstacles in the space around the vehicle based on the ultrasonic information; Obtaining image information captured by the vehicle-mounted camera assembly, and obtaining obstacle information of visual obstacles around the vehicle based on the image information; Synchronize obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time; Determine one or more target obstacle points according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and perform information fusion to output obstacle information of the one or more target obstacle points in a preset format; If any spatial obstacle and any visual obstacle refer to the same target obstacle, information fusion is performed based on the obstacle information of the spatial obstacle and the visual obstacle to obtain obstacle information of a set of obstacle points in a preset format representing the target obstacle; If any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is regarded as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle; If any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained based on the obstacle information of the visual obstacle.

2. The obstacle information fusion method of the intelligent parking system according to claim 1, characterized in that: Obtaining obstacle information of obstacles in the space around the vehicle according to the ultrasonic information includes: Calculating position information of multiple obstacle points relative to the center of the vehicle based on the ultrasonic information; According to the position information of the plurality of obstacle points relative to the center of the vehicle, the plurality of obstacle points are grouped to obtain one or more groups of obstacle points, and obstacle information of obstacles in the space around the vehicle is further obtained; Among them, multiple obstacle points in the same group represent multiple points on the same spatial obstacle; the obstacle information of each spatial obstacle includes a group number of a group of obstacle points and position information of all obstacle points in the group.

3. The obstacle information fusion method for an intelligent parking system according to claim 2, characterized in that: Obtaining obstacle information of visual obstacles around the vehicle according to the image information includes: The image information is processed by image recognition using a pre-trained deep learning network model to output obstacle information of visual obstacles around the vehicle; wherein the obstacle information of the visual obstacle includes the location information, width, category, category confidence and timestamp of the visual obstacle; the timestamp indicates the time when the on-board camera component captures the visual obstacle.

4. The obstacle information fusion method for an intelligent parking system according to claim 3, characterized in that: The step of synchronizing the obstacle information of the spatial obstacles and the visual obstacles around the vehicle in space and time includes: Mapping the position information of all obstacle points and visual obstacles to the same vehicle coordinate system, obtaining the coordinates of all obstacle points and visual obstacles in the same vehicle coordinate system, so as to achieve spatial synchronization; wherein the vehicle coordinate system is a two-dimensional Cartesian coordinate system established with the center of the vehicle as the origin; The compensation time of the visual obstacle is calculated according to the timestamp and the preset time compensation algorithm; the historical running status of the vehicle is obtained, and the coordinates of the visual obstacle in the vehicle coordinate system are updated according to the compensation time and the historical running status of the vehicle to achieve time synchronization.

5. The intelligent parking system obstacle information fusion method according to claim 4, characterized in that: in, The method for determining whether a spatial obstacle and a visual obstacle refer to the same target obstacle is as follows: In the vehicle coordinate system, taking the coordinates of any visual obstacle as the origin and 1 / 2 of the width of the visual obstacle as the radius, a circular area is established as the visual target to be matched area; The ratio of the number of obstacle points in each group of obstacle points located in the visual target to be matched area to the total number of obstacle points in the group of obstacle points is calculated respectively, and it is determined whether the largest ratio is greater than a preset threshold; if so, the spatial obstacle corresponding to the largest ratio and the visual obstacle refer to the same target obstacle; if not, there is no spatial obstacle that refers to the same target obstacle as the visual obstacle.

6. The obstacle information fusion method for an intelligent parking system according to claim 4, characterized in that: The obstacle information in the preset format includes the coordinates, category, and category confidence of the target obstacle; Wherein, the obtaining obstacle information of a set of obstacle points in a preset format representing the target obstacle by fusing the obstacle information of the spatial obstacle and the visual obstacle comprises: taking the coordinates of the set of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system as the coordinates of the set of obstacle points representing the target obstacle; taking the category and category confidence of the visual obstacle as the category and category confidence of the set of obstacle points representing the target obstacle; Wherein, obtaining obstacle information of a preset format of a group of obstacle points representing the target obstacle according to the obstacle information of the spatial obstacle includes: using coordinates of a group of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system as coordinates of a group of obstacle points representing the target obstacle; using "spatial obstacle" as the category of the group of obstacle points representing the target obstacle, and setting the category confidence to 100%; The obtaining, according to the obstacle information of the visual obstacle, obstacle information in a preset format representing an obstacle point of the target obstacle comprises: taking the coordinates of the visual obstacle in the vehicle coordinate system as the coordinates of an obstacle point representing the target obstacle; and taking the category and category confidence of the visual obstacle as the category and category confidence of an obstacle point representing the target obstacle.

7. An intelligent parking system obstacle information fusion system, characterized in that: include: A spatial obstacle detection unit, used to obtain ultrasonic information detected by the vehicle-mounted ultrasonic sensor assembly, and obtain obstacle information of spatial obstacles around the vehicle based on the ultrasonic information; A visual obstacle detection unit, used to obtain image information captured by the vehicle-mounted camera assembly, and obtain obstacle information of visual obstacles around the vehicle based on the image information; A space-time synchronization unit for synchronizing obstacle information of spatial obstacles and visual obstacles around the vehicle in space and time; as well as An information fusion unit, used to determine one or more target obstacle points according to the synchronized obstacle information of the spatial obstacles and visual obstacles around the vehicle, and perform information fusion to output the obstacle information of the one or more target obstacle points in a preset format; in: When any spatial obstacle and any visual obstacle refer to the same target obstacle, information fusion is performed based on the obstacle information of the spatial obstacle and the visual obstacle to obtain obstacle information of a set of obstacle points in a preset format representing the target obstacle; When any spatial obstacle does not have a visual obstacle that refers to the same target obstacle as the spatial obstacle, the spatial obstacle is regarded as a target obstacle, and obstacle information of a set of obstacle points in a preset format representing the target obstacle is obtained according to the obstacle information of the spatial obstacle; When any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as the visual obstacle, the visual obstacle is regarded as a target obstacle, and obstacle information in a preset format representing an obstacle point of the target obstacle is obtained according to the obstacle information of the visual obstacle.

8. The intelligent parking system obstacle information fusion system according to claim 7, characterized in that: The obstacle information in the preset format includes the coordinates, category, and category confidence of the target obstacle; When any spatial obstacle and any visual obstacle refer to the same target obstacle, the coordinates of a set of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a set of obstacle points representing the target obstacle; the category and category confidence of the visual obstacle are used as the category and category confidence of a set of obstacle points representing the target obstacle; Wherein, when there is no visual obstacle that refers to the same target obstacle as any spatial obstacle, the coordinates of a set of obstacle points corresponding to the spatial obstacle in the vehicle coordinate system are used as the coordinates of a set of obstacle points representing the target obstacle; "spatial obstacle" is used as the category of the set of obstacle points representing the target obstacle, and the category confidence is set to 100%; Among them, when any visual obstacle does not have a spatial obstacle that refers to the same target obstacle as it, the coordinates of the visual obstacle in the vehicle coordinate system are used as the coordinates of an obstacle point representing the target obstacle; and the category and category confidence of the visual obstacle are used as the category and category confidence of an obstacle point representing the target obstacle.

Images