Obstacle identification method and device

Abstract

The present invention provides an obstacle identification method and device, the method comprising: determining a point cloud target object based on point cloud data of a target track area; determining a track image and an image target object based on image data of a target track area that is closest to the acquisition time of the point cloud data; determining the position of the point cloud target object based on the point cloud target object and the track image; determining the type of the point cloud target object based on the point cloud target object and the image target object; and determining the position and type of an obstacle based on the position and type of the point cloud target object. The obstacle identification method and device provided by the present invention can improve the reliability of obstacle identification by performing obstacle identification mainly based on point cloud data and determining whether the point cloud target object is on the track by judging the relative position relationship between the point cloud target object and the track.

Description

Technical Field

[0001] The present invention relates to the field of rail transit technology, and in particular to an obstacle recognition method and device. Background Art

[0002] With the widespread application of artificial intelligence technology, obstacle perception technology based on deep learning has been widely used in fields such as rail transit, and the detection of vehicles, personnel and other targets in front can be achieved through deep learning technology of images.

[0003] Deep learning technology requires the collection of a large number of target images for training in advance. For some obstacles that have a low probability of occurrence and lack uniform features, such as concrete blocks, metal tools, steel plates, wooden sticks, etc., it is impossible to collect a large amount of data in advance and complete training. Therefore, deep learning technology can usually only detect a few common targets, such as vehicles, people, cats and dogs, etc. Summary of the invention

[0004] The present invention provides an obstacle recognition method and device, which are used to solve the defect that untrained obstacles cannot be recognized in the prior art, and realize reliable recognition of obstacles.

[0005] The present invention provides an obstacle recognition method, comprising:

[0006] Determine the point cloud target object according to the point cloud data of the target track area;

[0007] Determine a track image and an image target object based on image data of a target track region closest to the acquisition time of the point cloud data;

[0008] Determining a position of the point cloud target object according to the point cloud target object and the track image;

[0009] Determining the type of the point cloud target object according to the point cloud target object and the image target object;

[0010] The position and type of the obstacle are determined according to the position and type of the point cloud target object.

[0011] According to the obstacle recognition method provided by the present invention, determining the type of the point cloud target object according to the point cloud target object and the image target object includes:

[0012] Determining the type confidence of each of the point cloud target objects according to the height and width of each of the point cloud target objects;

[0013] Determining a type confidence of each of the image target objects based on the image data;

[0014] The type of the point cloud target object is determined according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object.

[0015] According to the obstacle identification method provided by the present invention, determining the type confidence of each point cloud target object according to the height and width of each point cloud target object includes:

[0016] Determine the aspect ratio confidence, height confidence and width confidence of each point cloud target object according to the height and width of each point cloud target object;

[0017] The type confidence of each of the point cloud target objects is determined according to the aspect ratio confidence, the height confidence and the width confidence.

[0018] According to the obstacle identification method provided by the present invention, determining the position of the point cloud target object according to the point cloud target object and the track image includes:

[0019] Projecting the point cloud data onto the image data to extract first point cloud data within the track image;

[0020] Extracting second point cloud data of the bottom portion of the point cloud target object or the image target object close to the ground;

[0021] Determine first ground point cloud data according to the first point cloud data and the second point cloud data;

[0022] Performing curve fitting on the first ground point cloud data to determine second ground point cloud data;

[0023] Projecting the second ground point cloud data into the image data to determine track point cloud data;

[0024] According to the track point cloud data and the point cloud target object, it is determined that the point cloud target object falls within a track area corresponding to the track point cloud data.

[0025] According to the obstacle recognition method provided by the present invention, determining the type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object, comprises:

[0026] When it is determined that the point cloud target object and the image target object are the same target object, the type of the point cloud target object is determined according to the type confidence of the point cloud target object and the type confidence of the image target object.

[0027] According to the obstacle recognition method provided by the present invention, the step of determining that the point cloud target object and the image target object are the same target object includes:

[0028] projecting the point cloud target object into the image data;

[0029] According to the degree of overlap between the point cloud target object and the image target object, it is determined that the point cloud target object and the image target object are the same target object.

[0030] The obstacle identification method provided by the present invention further includes:

[0031] Lock and track the target obstacle;

[0032] Wherein, the target obstacle is identified based on the point cloud data of a continuous target number of frames and the image data of a continuous target number of frames;

[0033] The position of the target obstacle is determined according to the movement trend of the target obstacle in the last frame of the point cloud data and the image data.

[0034] The present invention also provides an obstacle recognition device, comprising:

[0035] A first determination module is used to determine a point cloud target object according to point cloud data of a target track area;

[0036] A second determination module is used to determine the track image and the image target object based on the image data of the target track area closest to the acquisition time of the point cloud data;

[0037] A third determination module, used to determine the position of the point cloud target object according to the point cloud target object and the track image;

[0038] a fourth determination module, configured to determine a type of the point cloud target object according to the point cloud target object and the image target object;

[0039] The fifth determination module is used to determine the position and type of the obstacle according to the position and type of the point cloud target object.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned obstacle identification methods are implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned obstacle identification methods are implemented.

[0042] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above obstacle identification methods are implemented.

[0043] The obstacle identification method, device, electronic device and storage medium provided by the present invention mainly perform obstacle identification based on point cloud data, and determine whether the point cloud target object is on the track by judging the relative position relationship between the point cloud target object and the track. It does not rely on sample collection and model training of the target to be detected in advance. Even if an obstacle that has never been trained appears in the track area, it can be identified, thereby improving the reliability of obstacle identification. It does not rely on electronic maps, and it is not necessary to establish an electronic map in advance to perceive and detect obstacles in unknown track areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flowchart of the obstacle identification method provided by the present invention;

[0046] Figure 2 It is one of the schematic diagrams of applying the obstacle identification method provided by the present invention;

[0047] Figure 3 This is the second schematic diagram of the obstacle identification method provided by the present invention;

[0048] Figure 4 This is a third schematic diagram of the obstacle identification method provided by the present invention;

[0049] Figure 5 This is a fourth schematic diagram of the obstacle identification method provided by the present invention;

[0050] Figure 6 This is a fifth schematic diagram of the obstacle identification method provided by the present invention;

[0051] Figure 7 is a schematic structural diagram of an obstacle identification device provided by the present invention;

[0052] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Figure 1 Schematic diagram of the obstacle identification method provided by the present invention. Figure 1 The obstacle identification method provided by the present invention may include:

[0055] Step S110, determining a point cloud target object according to the point cloud data of the target track area;

[0056] Step S120, determining the track image and the image target object according to the image data of the target track area that is closest to the acquisition time of the point cloud data;

[0057] Step S130: Determine the position of the point cloud target object according to the point cloud target object and the track image;.

[0058] Step S140, determining the type of the point cloud target object according to the point cloud target object and the image target object;

[0059] Step S150: Determine the position and type of the obstacle according to the position and type of the point cloud target object.

[0060] It should be noted that the execution subject of the obstacle identification method provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited by the present invention.

[0061] Specifically, in step S110, a point cloud target object is determined based on the point cloud data of the target track area.

[0062] Point cloud data is a collection of sampling points with spatial coordinates obtained by LiDAR. LiDAR can be installed on trains, and when the train is running on the rails, the LiDAR can collect three-dimensional point cloud data around the track.

[0063] According to the collected point cloud data, point cloud target objects are determined through point cloud target detection. Specifically, point cloud clustering and segmentation technology can be used to detect the targets in the point cloud one by one to obtain point cloud target objects. The point cloud target objects may be one or more.

[0064] Point cloud target objects can be pedestrians, vehicles, concrete blocks, metal tools, steel plates, or wooden sticks, and any other objects that appear in the target area. However, point cloud target detection can only detect the presence of target objects in the collected area, which can be recorded as point cloud target object 1, point cloud target object 2, point cloud target object 3, etc. However, the specific type of the target object cannot be confirmed.

[0065] In addition, a unique ID can be assigned to each point cloud target object, and the point cloud target object can be continuously tracked according to the proximity principle, so as to determine the type of the point cloud target object in the subsequent process.

[0066] In step S120, the track image and the image target object are determined based on the image data of the target track area that is closest to the acquisition time of the point cloud data.

[0067] An image acquisition device may be installed on a train, and when the train is traveling on the rails, the image acquisition device may acquire image data around the rails. The image acquisition device may be a camera.

[0068] According to the data acquisition time, the image data with the closest acquisition time is found for each frame of point cloud data acquired in step S110, and the two can be matched into a pair. Because the target object may be in motion, for a moving vehicle, when the paired point cloud is projected onto the image, the deviation between the point cloud target object and the image target object can be guaranteed to be minimal.

[0069] Based on the collected image data, the track image in the image data is determined through image track detection. Specifically, deep learning technology can be used to detect the track or track area from the image. If there is a turnout, the type of turnout needs to be accurately detected, such as a separate turnout or a merging turnout. Complete track detection requires obtaining the "semantic level" information of the track. The semantic level means that if there are multiple tracks and turnouts in the scene, each track can be individually numbered and the turnout relationship between tracks can be identified, such as a merging turnout and a separate turnout.

[0070] According to the collected image data, the image target object in the image data is determined through image target detection, which can be specifically detected from the image data collected by the camera through deep learning technology, such as the front of the vehicle, the vehicle stalls, pedestrians and other typical obstacles. In step S130, the position of the point cloud target object is determined according to the point cloud target object and the track image.

[0071] Specifically, the results of image track detection can be projected into the three-dimensional space of the point cloud through track mapping, so that it is possible to know whether the position of the point cloud target object is within the track area of ​​the vehicle, and the point cloud target object within the track area can be determined as an obstacle.

[0072] In step S140 , the type of the point cloud target object is determined according to the point cloud target object and the image target object.

[0073] Through the joint calibration parameters of image data and point cloud data, the point cloud target object is projected onto the image data, and then the point cloud target and the image target are matched one by one. Once matched, the corresponding relationship between the point cloud target object and the image target object can be found, and the results of point cloud target recognition and image target recognition are summarized, and the point cloud target object has a certain type.

[0074] In step S150, the position and type of the obstacle are determined according to the position and type of the point cloud target object.

[0075] If the point cloud target object is located within the track area, it can be identified as an obstacle. The position and type of the obstacle in the target area can be known based on the position and type of the point cloud target object.

[0076] The obstacle recognition method provided by the embodiment of the present invention mainly recognizes obstacles based on point cloud data, and determines whether the point cloud target object is on the track by judging the relative position relationship between the point cloud target object and the track. It does not rely on sample collection and model training of the target to be detected in advance. Even if an obstacle that has never been trained appears in the track area, it can be recognized, thereby improving the reliability of obstacle recognition. It does not rely on electronic maps, and it is not necessary to establish an electronic map in advance to perceive and detect obstacles in unknown track areas.

[0077] In one embodiment, the obstacle recognition method provided by the present invention determines the type of the point cloud target object according to the point cloud target object and the image target object, including:

[0078] Determine the type confidence of each point cloud target object according to the height and width of each point cloud target object;

[0079] Determine the type confidence of each image target object based on the image data;

[0080] The type of the point cloud target object is determined according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object.

[0081] Specifically, type confidence refers to the probability that a point cloud target object is of a certain type, such as the probability that a point cloud target object is a vehicle or a pedestrian. Based on the point cloud target objects detected by the point cloud target, the height and width of each point cloud target object can be obtained, and the confidence of whether it is a vehicle, a person, etc. can be calculated based on the feature information such as the aspect ratio and size of each target. For example, if the height and width of a point cloud target object are very close to the height and width of a vehicle, it can be considered that the confidence of the point cloud target object being a vehicle is relatively high.

[0082] Image object detection refers to the use of deep learning technology to detect typical obstacles such as vehicle heads, vehicle stops, and pedestrians from images. All deep learning object detection models will output a target confidence C while outputting the target detection frame. c (0 <C c <1) (where c represents camera).

[0083] The type of the point cloud target object is obtained by combining the detection results of the point cloud target object determined by the point cloud data and the confidence of each point cloud target object, and the image target object determined by the image data and the type confidence of each image target object.

[0084] The obstacle recognition method provided in the embodiment of the present invention can effectively identify the type of obstacles and ensure the reliability of recognition by combining point cloud target detection and image target detection to determine the type confidence of the target object respectively, and combining the results of point cloud target detection and image target detection.

[0085] In one embodiment, determining the type confidence of each point cloud target object according to the height and width of each point cloud target object includes:

[0086] According to the height and width of each point cloud target object, determine the aspect ratio confidence, height confidence and width confidence of each point cloud target object;

[0087] The type confidence of each point cloud target object is determined based on the aspect ratio confidence, the height confidence and the width confidence.

[0088] Specifically, it is known that the standard height of the front of a subway train is h S Meters, standard width is w S Meters, aspect ratio is r S The height-to-width ratio r, height h, and width w of the point cloud target object detected by the point cloud target, then the confidence of the vehicle head type (where l means radar and T means train) can be determined by:

[0089] First, calculate the aspect ratio confidence c of the point cloud target object r , high confidence c h , width confidence c w , respectively:

[0090]

[0091]

[0092]

[0093] Then calculate the total confidence after superposition of the three That is, the confidence that the point cloud target object is a vehicle is:

[0094]

[0095] The obstacle recognition method provided by the embodiment of the present invention can improve the accuracy of obstacle type recognition by calculating the type confidence of the point cloud target object by comprehensively considering the height, width and aspect ratio of the point cloud target object.

[0096] In one embodiment, determining the position of the point cloud target object according to the point cloud target object and the track image includes:

[0097] Projecting the point cloud data onto the image data, extracting first point cloud data within the track image;

[0098] Extracting second point cloud data of the bottom portion of the point cloud target object or the image target object close to the ground;

[0099] Determine first ground point cloud data according to the first point cloud data and the second point cloud data;

[0100] Performing curve fitting on the first ground point cloud data to determine second ground point cloud data;

[0101] Projecting the second ground point cloud data into the image data to determine the track point cloud data;

[0102] According to the track point cloud data and the point cloud target object, it is determined that the point cloud target object falls within the track area corresponding to the track point cloud data.

[0103] Specifically, LiDAR can only detect short-distance tracks, but has no detection capability for long-distance tracks, which means that it is impossible to determine whether a long-distance target is on the track simply by relying on LiDAR. Therefore, the detection of long-distance tracks depends largely on image track detection, and then the results of image track detection are projected into the three-dimensional space of the point cloud, so that it can be determined whether the point cloud target is within the track area of ​​the vehicle.

[0104] The main problem in projecting the results of image track detection into three-dimensional space is how to obtain the height of the ground, especially the height of the ground at the far end. The following steps can be used to project the results of image track detection into three-dimensional space.

[0105] Step 1: Take a pair of point cloud data and image data, use the joint calibration parameters to completely project the point cloud data onto the image data, and then extract the point cloud that falls within the image track detection frame based on the results of the image track detection to obtain the track point cloud of the close-range section, which is the first point cloud data. Figure 2 is one of the schematic diagrams of the obstacle identification method provided by the present invention, referring to Figure 2 , extract the track point cloud data in the image track detection frame. The track point cloud usually only has the close-range part and lacks the long-range part.

[0106] Step 2: Extract the point cloud corresponding to pedestrians, vehicles and other targets in the distance close to the image track detection frame as much as possible, and take the point cloud of the bottom part close to the ground, which is the second point cloud data. If no pedestrians or vehicles are detected in the image, point cloud targets detected can also be used, even if the point cloud targets are not recognized, which are mainly used for remote ground height estimation. Figure 3 This is a second schematic diagram of the obstacle identification method provided by the present invention, referring to Figure 3 ,like Figure 3 As shown in the top view in the figure, a total of two targets were found, one is the vehicle in front and the other is a pedestrian on the trackside.

[0107] Step 3: Put the point clouds obtained in step 1 and step 2 into the same three-dimensional coordinate system to obtain discontinuous ground point clouds from near distance to far end, that is, the first ground point cloud data, such as Figure 3 As shown in the middle side view, the track point cloud at close distance is still relatively dense, but as the distance increases, the track point cloud becomes increasingly sparse until it disappears completely. At a longer distance, the ground height can only be obtained by relying on the bottom point cloud of pedestrians and the vehicle in front.

[0108] Step 4: Perform quadratic or cubic curve fitting on the ZY plane for the discontinuous ground point cloud obtained in step 3 to obtain the height calculation formula z=ay at different distances of the ground 2+by+c, where the three parameters a, b, and c are obtained through a curve fitting algorithm.

[0109] Step 5: According to the result of step 4, construct a virtual ground point cloud with equal spacing, that is, the second ground point cloud data. Figure 4 This is a third schematic diagram of the obstacle identification method provided by the present invention, referring to Figure 4 ,like Figure 4 As shown in the top view, the constructed virtual point cloud appears in a grid shape from the top view, and the point spacing is fixed, such as 10 cm. The z coordinate and y coordinate of each point must satisfy the formula z = ay 2 +by+c, such as Figure 4 As shown in the side view, the virtual ground point cloud completely coincides with the ground curve obtained in step 4.

[0110] Step 6: Use the joint calibration parameters to project the virtual ground point cloud obtained in step 5 onto the image again, and then extract the virtual ground point cloud that falls within the image track detection frame based on the results of the image track detection to obtain the virtual track point cloud, which is the track point cloud data required in the end. Unlike the track point cloud extracted in step 1, the virtual track point cloud does not have the problem of being too close in distance, and has points evenly distributed from near to far.

[0111] Step 7: Place the virtual track point cloud and the detected target point cloud in the same three-dimensional coordinate system, and check whether the target point cloud falls within the track area corresponding to the virtual track point cloud. If so, it is determined to be an obstacle that needs to be protected, otherwise it is not. Figure 5 FIG. 4 is a schematic diagram of the obstacle identification method provided by the present invention. Figure 5 As shown, the pedestrian point cloud is outside the track area and is not an obstacle, while the vehicle point cloud falls completely within the track area and is an obstacle that needs to be protected.

[0112] In addition to the above methods for estimating ground height, there are other methods for projecting the track area into three-dimensional space, such as monocular visual depth estimation based on deep learning technology, three-dimensional reconstruction based on VSLAM technology, and so on.

[0113] After the track mapping is completed, it is clear whether the point cloud target object is on the track area, so the point cloud target object on the track area is determined as an obstacle.

[0114] The obstacle recognition method provided by the embodiment of the present invention transforms the track detected by the image into a bird's-eye view or three-dimensional space by utilizing ground height estimation and affine transformation, and then superimposes the target detected by the point cloud target into the same bird's-eye view or three-dimensional space, so as to determine whether the target is in the track area of ​​the vehicle. If it is in the track area, it is an obstacle, otherwise it is not. At the same time, the problem that long-distance tracks cannot be detected by laser radar alone is solved, and the reliable detection of long-distance obstacles is significantly improved.

[0115] In one embodiment, determining the type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object includes:

[0116] When it is determined that the point cloud target object and the image target object are the same target object, the type of the point cloud target object is determined according to the type confidence of the point cloud target object and the type confidence of the image target object.

[0117] Specifically, the target object according to the point cloud is a three-dimensional object, and the target object according to the image is a two-dimensional object. First, it is necessary to determine whether the identified point cloud target object and the image target object are the same target object, that is, to match the point cloud target object with the image target object one by one. Then, the type recognition results of the point cloud and image target detection are integrated, and the recognition results include type and confidence, to obtain a more reliable result.

[0118] For example, both point cloud target detection and image target detection identify the type of a certain target (such as the front of a train) and the corresponding confidence levels are and Then, the two confidence levels are combined by the following method to obtain the combined confidence level C: T :

[0119]

[0120] Where k is the confidence ratio factor of radar and camera.

[0121] If the confidence of the radar is considered higher, k is selected in the interval (0.5, 1), otherwise in the interval (0, 0.5). If the confidence of the two is considered to be the same, 0.5 is directly selected. If only the camera detects and identifies the type, the above formula is still used for calculation, but the confidence is lower than that of simultaneous detection.

[0122] The obstacle recognition method provided by the embodiment of the present invention obtains a reliable result of the obstacle type and its confidence level by integrating the detection results of the point cloud target object and the image target object (each with the obstacle type and its confidence level) through the correspondence between the point cloud target object and the image target object. Even if the result of the image target detection is not very reliable, a certain degree of missed reports and false reports will not affect the reliable recognition of the target object type.

[0123] In one embodiment, determining that the point cloud target object and the image target object are the same target object includes:

[0124] Projecting the point cloud target object into the image data;

[0125] According to the degree of coincidence between the point cloud target object and the image target object, it is determined that the point cloud target object and the image target object are the same target object.

[0126] Specifically, determining that the point cloud target object and the image target object are the same target object means projecting the point cloud target onto the image through the joint calibration parameters of the image and the point cloud, and then matching the point cloud target object with the image target object one by one. Once matched, the corresponding relationship between the point cloud target object and the image target object can be found, and the results of point cloud target recognition and image target recognition are summarized, and the point cloud target object has a determined type.

[0127] Before point cloud projection, the laser radar and camera must be jointly calibrated to obtain the camera's joint calibration parameters. Then, the paired point cloud data and image data are taken, and the calibration parameters are used to project the 3D point cloud onto the 2D plane of the image, so that the point cloud of the same object completely overlaps with the image contour.

[0128] The fundamental purpose of joint calibration of LiDAR and camera is to project the point cloud of LiDAR onto the image so that the point cloud of the same object completely overlaps with the image contour. To achieve this goal, two steps are required:

[0129] Step 1: Use external parameters to transform the points in the point cloud from the radar coordinate system (x w ,y w ,z w ) is transformed into the camera coordinate system (x c ,y c , z c ):

[0130]

[0131] Where R is the rotation matrix, which can also be expressed by the rotation in three directions (Pitch, Roll, Yaw) in the rectangular coordinate system, and t is the translation vector, that is, the distance in three directions in the rectangular coordinate system (t x ,t y , tz ), rotation plus translation, a total of six parameters, constitute the external parameters.

[0132] Step 2: Use the intrinsic parameter to transform the point (x c ,y c , z c ) into pixel coordinates (u,v):

[0133]

[0134] where f x and f y is the focal length in the horizontal and vertical directions, u0 and v0 are the pixel coordinates of the camera center, and s is the axis tilt coefficient. A total of five parameters constitute the internal parameters.

[0135] The intrinsic parameters are only related to the camera (body and lens). The industry has a common and mature detection algorithm to obtain the camera's intrinsic parameters. The conventional method is to use a camera to take several frames of checkerboard photos in different postures, and then input the photos into a standard detection algorithm to output the internal parameters.

[0136] The external parameters are only related to the relative position relationship between the radar and the camera. It is usually necessary to project the point cloud onto the image based on the measured internal parameters and a set of empirically determined external parameter estimates, adjust the external parameters so that the point cloud of the same object completely overlaps with the image contour, and then obtain the final external parameters.

[0137] After obtaining the joint calibration parameters, the target detected by the point cloud target is projected onto the image and matched with the result of the image target detection, and the degree of overlap is used to determine whether the two are the same target.

[0138] Figure 6 FIG. 5 is a schematic diagram of the obstacle identification method provided by the present invention, referring to FIG. Figure 6 , we can get: the detection results of the front of the car in the figure completely overlap, so it can be determined that the two are the same target. The point cloud detection frame and the image detection frame of person B do not completely overlap, so it is necessary to calculate the intersection over union (IOU), that is, the ratio of the intersection and union of the two detection frames. The threshold is usually set to 0.5, that is, when IOU>0.5, the two are considered to be the same target, otherwise they are not the same target. Person A and the traffic light are only detected by point cloud, not image, so it can only be confirmed that there is a target object at this location, but the specific type cannot be verified.

[0139] The obstacle recognition method provided by the embodiment of the present invention jointly calibrates the laser radar and the camera, projects the point cloud target onto the image, and determines whether the two target objects are the same target object based on the degree of overlap between the point cloud target object and the image target object, thereby improving the accuracy of obstacle type recognition.

[0140] In one embodiment, the obstacle identification method provided by the present invention further includes:

[0141] Lock and track the target obstacle;

[0142] Wherein, the target obstacle is identified based on point cloud data of a continuous target number of frames and image data of a continuous target number of frames;

[0143] The position of the target obstacle is determined according to the movement trend of the target obstacle in the last frame of point cloud data and image data.

[0144] Specifically, once obstacles such as vehicles and pedestrians appear in front, they will not suddenly disappear, but the detection results of the sensor may jitter or be missed, especially image target detection based on deep learning, which cannot completely guarantee that obstacles can be reliably detected at all times. The reliability of lidar is relatively high, and the stability of detection is guaranteed, but if there is occlusion, missed detection may occur.

[0145] In order to avoid missed detection, it is necessary to lock and track the obstacle. Once the obstacle is locked, even if it is not detected in several frames of data, the system will continue to report the obstacle, and estimate the position of the obstacle based on the movement trend when it was detected in the last frame of data, and update the position and distance of the obstacle.

[0146] Especially for large targets such as vehicles, their running speed and direction are unlikely to change much in a short period of time. By locking and tracking, a more accurate position estimate can be given.

[0147] The prerequisite for entering locked tracking is that several consecutive frames are detected and it is determined that they are in the tracking area or near the tracking area.

[0148] For large targets such as vehicles, an object needs to be established in the system to record its position, speed and direction of movement at all times, and update its position and speed based on the results of each frame detection. When a missed detection occurs, the position is estimated and reported based on its position and speed at the last moment. Compared with small targets such as pedestrians, the tolerance for missed detection is higher, that is, even if there is a missed detection for multiple frames, obstacles will continue to be reported.

[0149] For small, low-speed targets such as pedestrians, an object also needs to be established in the system to record their position, speed, and direction of movement at all times, and update their position and speed based on the results of each frame detection. When a missed detection occurs, no position estimation is performed, and obstacles are continuously reported based on the position at the last moment. Compared with large targets such as vehicles, the tolerance for missed detection is lower, and the lock tracking is canceled when no detection is performed for several frames.

[0150] The obstacle recognition method provided by the embodiment of the present invention can effectively avoid missed detection by locking and tracking the obstacle, and determine the specific position of the obstacle according to the position and speed of the obstacle in the last frame of data, thereby ensuring the reliability of obstacle recognition.

[0151] The obstacle recognition device provided by the present invention is described below. The obstacle recognition device described below and the obstacle recognition method described above can be referred to each other.

[0152] Figure 7 A schematic diagram of the structure of the obstacle recognition device provided by the present invention is shown in FIG. Figure 7 As shown, the device may include:

[0153] A first determination module 710 is used to determine a point cloud target object according to point cloud data of a target track area;

[0154] A second determination module 720 is used to determine the track image and the image target object based on the image data of the target track area that is closest to the acquisition time of the point cloud data;

[0155] A third determination module 730 is used to determine the position of the point cloud target object according to the point cloud target object and the track image;

[0156] A fourth determination module 740, configured to determine the type of the point cloud target object according to the point cloud target object and the image target object;

[0157] The fifth determination module 750 is used to determine the position and type of the obstacle according to the position and type of the point cloud target object.

[0158] The obstacle recognition device provided by the embodiment of the present invention recognizes obstacles mainly based on point cloud data, and determines whether the point cloud target object is on the track by judging the relative position relationship between the point cloud target object and the track. It does not rely on sample collection and model training of the target to be detected in advance. Even if an obstacle that has never been trained appears in the track area, it can be recognized, thereby improving the reliability of obstacle recognition. It does not rely on electronic maps, and it is not necessary to establish an electronic map in advance to perceive and detect obstacles in unknown track areas.

[0159] In one embodiment, the fourth determination module 740 is specifically used to:

[0160] Determine the type confidence of each point cloud target object according to the height and width of each point cloud target object;

[0161] Determine the image target object and the type confidence of each image target object according to the image data;

[0162] The obstacle type of the point cloud target object is determined according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object.

[0163] In one embodiment, determining the type confidence of each point cloud target object according to the height and width of each point cloud target object includes:

[0164] According to the height and width of each point cloud target object, determine the aspect ratio confidence, height confidence and width confidence of each point cloud target object;

[0165] The type confidence of each point cloud target object is determined based on the aspect ratio confidence, the height confidence and the width confidence.

[0166] In one embodiment, the third determination module 730 is specifically configured to:

[0167] Projecting the point cloud data onto the image data, extracting first point cloud data within the track image;

[0168] Extracting second point cloud data of the bottom portion of the point cloud target object or the image target object close to the ground;

[0169] Determine first ground point cloud data according to the first point cloud data and the second point cloud data;

[0170] Performing curve fitting on the first ground point cloud data to determine second ground point cloud data;

[0171] Projecting the second ground point cloud data into the image data to determine the track point cloud data;

[0172] According to the track point cloud data and the point cloud target object, the point cloud target object falling within the track area corresponding to the track point cloud data is determined as an obstacle.

[0173] In one embodiment, determining the obstacle type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object includes:

[0174] When it is determined that the point cloud target object and the image target object are the same target object, the obstacle type of the point cloud target object is determined according to the type confidence of the point cloud target object and the type confidence of the image target object.

[0175] In one embodiment, determining that the point cloud target object and the image target object are the same target object includes:

[0176] Projecting the point cloud target object into the image data;

[0177] According to the degree of coincidence between the point cloud target object and the image target object, it is determined that the point cloud target object and the image target object are the same target object.

[0178] In one embodiment, the obstacle identification device provided by the embodiment of the present invention further includes:

[0179] Tracking module, used to lock and track the target obstacle;

[0180] Wherein, the target obstacle is identified based on point cloud data of a continuous target number of frames and image data of a continuous target number of frames;

[0181] The position determination module is used to determine the position of the target obstacle according to the movement trend of the target obstacle in the last frame of point cloud data and image data.

[0182] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the obstacle recognition method, which includes:

[0183] Determine the point cloud target object according to the point cloud data of the target track area;

[0184] Determine the track image and the image target object based on the image data of the target track area closest to the acquisition time of the point cloud data;

[0185] Determine the position of the point cloud target object according to the point cloud target object and the track image;

[0186] Determine the type of the point cloud target object according to the point cloud target object and the image target object;

[0187] The location and type of obstacles are determined based on the location and type of the point cloud target object.

[0188] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0189] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the obstacle identification method provided by the above methods, the method includes:

[0190] Determine the point cloud target object according to the point cloud data of the target track area;

[0191] Determine the track image and the image target object based on the image data of the target track area closest to the acquisition time of the point cloud data;

[0192] Determine the position of the point cloud target object according to the point cloud target object and the track image;

[0193] Determine the type of the point cloud target object according to the point cloud target object and the image target object;

[0194] The location and type of obstacles are determined based on the location and type of the point cloud target object.

[0195] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the obstacle identification method provided by the above methods is implemented, and the method includes:

[0196] Determine the point cloud target object according to the point cloud data of the target track area;

[0197] Determine the track image and the image target object based on the image data of the target track area closest to the acquisition time of the point cloud data;

[0198] Determine the position of the point cloud target object according to the point cloud target object and the track image;

[0199] Determine the type of the point cloud target object according to the point cloud target object and the image target object;

[0200] The location and type of obstacles are determined based on the location and type of the point cloud target object.

[0201] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An obstacle recognition method, characterized in that: include: Determine the point cloud target object according to the point cloud data of the target track area; Determine a track image and an image target object based on image data of a target track region closest to the acquisition time of the point cloud data; Determining a position of the point cloud target object according to the point cloud target object and the track image; Determining the type of the point cloud target object according to the point cloud target object and the image target object; Determining the location and type of the obstacle according to the location and type of the point cloud target object; Wherein, determining the type of the point cloud target object according to the point cloud target object and the image target object includes: According to the height and width of each point cloud target object and the height and width of the subway train head, the aspect ratio confidence, height confidence and width confidence of each point cloud target object are determined by the following formula: Determining the type confidence of each of the point cloud target objects according to the aspect ratio confidence, the height confidence and the width confidence; Determining a type confidence of each of the image target objects based on the image data; Determining the type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object; Among them, the c r is the aspect ratio confidence of the point cloud target object, the c h is the high confidence of the point cloud target object, the c w is the width confidence of the point cloud target object, and the r s is the height-to-width ratio of the subway train head, and the h s is the standard height of the subway train head, and the w s is the standard width of the front of the subway train, r is the aspect ratio of the point cloud target object, h is the height of the point cloud target object, and w is the width of the point cloud target object.

2. The obstacle recognition method according to claim 1, characterized in that: The step of determining the position of the point cloud target object according to the point cloud target object and the track image comprises: Projecting the point cloud data onto the image data to extract first point cloud data within the track image; Extracting second point cloud data of the bottom portion of the point cloud target object or the image target object close to the ground; Determine first ground point cloud data according to the first point cloud data and the second point cloud data; Performing curve fitting on the first ground point cloud data to determine second ground point cloud data; Projecting the second ground point cloud data into the image data to determine track point cloud data; According to the track point cloud data and the point cloud target object, it is determined that the point cloud target object falls within a track area corresponding to the track point cloud data.

3. The obstacle recognition method according to claim 1, characterized in that: The determining the type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object, comprises: When it is determined that the point cloud target object and the image target object are the same target object, the type of the point cloud target object is determined according to the type confidence of the point cloud target object and the type confidence of the image target object.

4. The obstacle recognition method according to claim 3, characterized in that: The determining that the point cloud target object and the image target object are the same target object includes: projecting the point cloud target object into the image data; According to the degree of overlap between the point cloud target object and the image target object, it is determined that the point cloud target object and the image target object are the same target object.

5. The obstacle recognition method according to claim 2, characterized in that: Also includes: Lock and track the target obstacle; Wherein, the target obstacle is identified based on the point cloud data of a continuous target number of frames and the image data of a continuous target number of frames; The position of the target obstacle is determined according to the movement trend of the target obstacle in the last frame of the point cloud data and the image data.

6. An obstacle recognition device, characterized in that: include: A first determination module is used to determine a point cloud target object according to point cloud data of a target track area; A second determination module is used to determine the track image and the image target object based on the image data of the target track area closest to the acquisition time of the point cloud data; A third determination module, configured to determine a position of the point cloud target object according to the point cloud target object and the track image; a fourth determination module, configured to determine a type of the point cloud target object according to the point cloud target object and the image target object; a fifth determination module, configured to determine the position and type of the obstacle according to the position and type of the point cloud target object; Wherein, the fourth determination module is specifically used for: According to the height and width of each point cloud target object and the height and width of the subway train head, the aspect ratio confidence, height confidence and width confidence of each point cloud target object are determined by the following formula: Determining the type confidence of each of the point cloud target objects according to the aspect ratio confidence, the height confidence and the width confidence; Determining a type confidence of each of the image target objects based on the image data; Determining the type of the point cloud target object according to the point cloud target object and the type confidence of the point cloud target object, and the image target object and the type confidence of the image target object; Among them, the c r is the aspect ratio confidence of the point cloud target object, the c h is the high confidence of the point cloud target object, the c w is the width confidence of the point cloud target object, and the r s is the height-to-width ratio of the subway train head, and the h s is the standard height of the subway train head, and the w s is the standard width of the front of the subway train, r is the aspect ratio of the point cloud target object, h is the height of the point cloud target object, and w is the width of the point cloud target object.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the obstacle recognition method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the obstacle identification method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the obstacle identification method according to any one of claims 1 to 5 are implemented.

Images