A Road Obstacle Perception System for Driverless Vehicles Based on Video Surveillance

Abstract

The present invention relates to the field of image communication technology, and particularly relates to an unmanned vehicle road obstacle perception system based on video surveillance. By acquiring road videos and obstacle distance information, the road videos are continuously tracked and detected through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images, and the motion law analysis is carried out according to the obstacle distance information to determine the consistency of the motion characteristics of obstacles in adjacent frame video images. When the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process, the occlusion degree of the obstacles is determined according to the appearance matching degree and the consistency of the motion characteristics, which is beneficial to improving the detection accuracy of obstacles when there is multi-obstacle associated interference. According to the occlusion degree of the obstacles, the tracking accuracy between obstacles in adjacent frame video images is determined and the position perception of the obstacles is carried out, which is beneficial to positioning and perceiving the obstacles when the obstacles are occluded.

Description

Technical Field

[0001] The present invention relates to the technical field of image communication, and particularly relates to an unmanned vehicle road obstacle perception system based on video surveillance. Background Art

[0002] An unmanned vehicle needs to have the function of identifying and perceiving obstacles (including pedestrians, vehicles, roadblocks, etc.), and realizing detection, classification, positioning and dynamic behavior prediction. Usually, the unmanned vehicle uses a camera to capture dynamic video data of the road scene in real time, which provides an information source for the detection and recognition of obstacles. Then, deep learning and computer vision algorithms are used to realize road obstacle perception. For example, an object tracking algorithm is used to perform multi-frame tracking on the obstacles, and the motion trajectory is predicted through the association matching of the obstacles in adjacent frame images.

[0003] However, the environment around an unmanned vehicle is usually dynamic, and the types and positions of obstacles change over time. In complex traffic flows, the motion patterns of other traffic participants (such as vehicles, pedestrians, etc.) may cause changes in the relative positions of obstacles, which easily leads to multi-obstacle association interference and makes multi-frame tracking difficult. Or, the obstacles may be partially occluded by different objects at different time points, resulting in the failure of the association matching of the obstacles in adjacent frame images, thus affecting obstacle perception, making it impossible to perform effective tracking, and unable to provide reliable support for the path planning and safe driving of the unmanned vehicle. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide an unmanned vehicle road obstacle perception system based on video surveillance, and the specific technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides an unmanned vehicle road obstacle perception system based on video surveillance, including:

[0006] An acquisition module, configured to acquire road videos through a camera of the unmanned vehicle and acquire obstacle distance information around the unmanned vehicle through a lidar;

[0007] A first determination module, configured to continuously track and detect the road video through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images, and perform motion law analysis according to the obstacle distance information to determine the motion feature consistency of obstacles in adjacent frame video images;

[0008] A second determination module, configured to determine the occlusion degree of an obstacle according to the appearance matching degree and the motion feature consistency when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process;

[0009] A perception module, configured to determine the tracking accuracy between obstacles in adjacent-frame video images according to the occlusion degree of the obstacles, and perform position perception of the obstacles according to the tracking accuracy.

[0010] In an implementation manner, the first determination module includes a first unit, a second unit, a third unit, and a fourth unit:

[0011] The first unit is configured to continuously track and detect the road video through an object tracking algorithm, and determine the edges of each obstacle in each frame of video image;

[0012] The second unit is configured to determine the consistency of the edge degree change between the obstacles in adjacent-frame video images according to the edges of the obstacles in each frame of video image, and determine the geometric feature consistency between the obstacles in adjacent-frame video images according to the consistency of the edge degree change;

[0013] The third unit is configured to determine the color feature consistency between the obstacles in adjacent-frame video images according to the edges of the obstacles in each frame of video image;

[0014] The fourth unit is configured to determine the appearance matching degree between the obstacles in adjacent-frame video images according to the first product of the geometric feature consistency and the color feature consistency.

[0015] In an implementation manner, the determining the color feature consistency between the obstacles in adjacent-frame video images according to the edges of the obstacles in each frame of video image includes:

[0016] Determine a first frame of video image from a plurality of frames of video images, and determine a second frame of video image that is the previous frame of the first frame of video image;

[0017] Respectively determine the first quantity of all pixel points and the second quantity corresponding to pixel points of different colors within the bounding boxes corresponding to the edges of each obstacle in the first frame of video image, and determine the third quantity corresponding to pixel points of different colors within the bounding boxes corresponding to the edges of each obstacle in the second frame of video image;

[0018] Respectively determine the number of color types included in the obstacles, and determine the color feature consistency between each obstacle in the first frame of video image and each obstacle in the second frame of video image according to the first quantity, the second quantity, the third quantity, and the number of color types, and return to the step of determining the first frame of video image from the plurality of frames of video images until the color feature consistency between the obstacles in adjacent-frame video images is determined.

[0019] In one embodiment, the first determination module further includes a fifth unit and a sixth unit:

[0020] The fifth unit is configured to determine the movement trend of each obstacle in each frame of video image according to the obstacle distance information corresponding to each frame of video image;

[0021] The sixth unit is configured to determine the consistency of the movement characteristics of each obstacle in adjacent frames of video images according to the movement trend of each obstacle in each frame of video image.

[0022] In one embodiment, the determining the movement trend of each obstacle in each frame of video image according to the obstacle distance information corresponding to each frame of video image includes:

[0023] Determine a third frame of video image from several frames of video images, determine several consecutive fourth frames of video images within a preset proximity range of the third frame of video image, and generate a position fitting straight line according to the obstacle distance information corresponding to the several consecutive fourth frames of video images;

[0024] According to the obstacle distance information corresponding to each frame of video image, respectively determine the first position of each obstacle in any one of the fourth frames of video images and the second position of the corresponding matching obstacle in the previous frame of the fourth frame of video images, and respectively determine the angle between the straight line connecting the first position and the second position and the corresponding position fitting straight line;

[0025] Respectively determine the velocity variances of each obstacle in the third frame of video image and each obstacle in each of the fourth frames of video images;

[0026] Obtain the average value of the angles by summing and averaging according to the angle and the number of the angles, and respectively determine the movement trend of each obstacle in the third frame of video image according to the second product of the average value of the angles and the velocity variances of the respective obstacles and the natural exponential function, and return to the step of determining the third frame of video image from several frames of video images until the movement trend of each obstacle in each frame of video image is determined.

[0027] In one embodiment, the determining the consistency of the movement characteristics of each obstacle in adjacent frames of video images according to the movement trend of each obstacle in each frame of video image includes:

[0028] Determine the movement trend differences of each obstacle in adjacent frames of video images according to the movement trend of each obstacle in each frame of video image;

[0029] Respectively determine the consistency of the movement characteristics of each obstacle in adjacent frames of video images according to the opposite number of the movement trend differences and the natural exponential function.

[0030] In one embodiment, the second determination module includes a seventh unit, an eighth unit, a ninth unit, and a tenth unit:

[0031] The seventh unit is configured to, when obstacles in adjacent frame video images cannot be correspondingly matched during continuous tracking detection, determine a fifth frame video image from a plurality of frame video images, and determine a sixth frame video image that is the previous frame of the fifth frame video image;

[0032] The eighth unit is configured to predict the positions of each obstacle in the sixth frame video image in the fifth frame video image through a uniformly accelerated motion model, to obtain a plurality of predicted positions;

[0033] The ninth unit is configured to determine the average appearance matching degree corresponding to each obstacle according to the appearance matching degree between each obstacle in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image, and respectively determine the first difference value between the appearance matching degree between each obstacle in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image and the average appearance matching degree corresponding to each obstacle;

[0034] The tenth unit is configured to respectively determine the occlusion degree of the obstacles at each of the predicted positions in the fifth frame video image according to the third product of the motion feature consistency of each obstacle in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image and the first difference value, and return the step of determining the fifth frame video image from a plurality of frame video images when obstacles in adjacent frame video images cannot be correspondingly matched during continuous tracking detection, until the occlusion degree of the obstacles at each of the predicted positions in each frame video image is determined.

[0035] In one embodiment, the perception module includes a first processing unit, a second processing unit, and a third processing unit:

[0036] The first processing unit is configured to determine the identity between each obstacle in adjacent frame video images according to the fourth product of the motion feature consistency and the appearance matching degree;

[0037] The second processing unit is configured to determine the corrected matching degree between the obstacles at each predicted position in each frame video image and the obstacles in the frame video image that is the previous frame of this frame video image according to the identity between each obstacle in the adjacent frame video images and the occlusion degree of the obstacles;

[0038] The third processing unit is configured to determine the tracking accuracy between each obstacle in adjacent frame video images according to the corrected matching degree.

[0039] In one embodiment, determining the corrected matching degree between the obstacles at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image according to the identity between the obstacles in the adjacent frame video images and the occlusion degree of the obstacles includes:

[0040] Respectively determine the sum value of a preset value and the occlusion degree of the obstacles at each predicted position in each frame of video image;

[0041] Respectively determine the fifth product according to the sum value and the identity between the obstacles in the adjacent frame video images, and determine the corrected matching degree between the obstacles at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image.

[0042] In one embodiment, determining the tracking accuracy between the obstacles in the adjacent frame video images according to the corrected matching degree includes:

[0043] According to the corrected matching degree between the obstacles at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image, respectively determine the average value of the corrected matching degrees corresponding to each obstacle;

[0044] Respectively determine the second difference value between the corrected matching degree between the obstacles at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image and the average value of the corrected matching degrees corresponding to each obstacle;

[0045] Respectively determine the calculation result according to the opposite number of the second difference value and the natural exponential function, and respectively determine the tracking accuracy between the obstacles in the adjacent frame video images according to the calculation result and the sixth product of the average value of the corrected matching degrees corresponding to each obstacle.

[0046] The present invention has the following beneficial effects:

[0047] Obtain road videos through the cameras of driverless vehicles and obtain the distance information of obstacles around the driverless vehicles through lidar. Continuously track and detect the road videos through object tracking algorithms to determine the appearance matching degree between obstacles in adjacent frame video images, and analyze the motion laws according to the obstacle distance information to determine the consistency of the motion characteristics of obstacles in adjacent frame video images. When the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process, determine the occlusion degree of the obstacles according to the appearance matching degree and the consistency of the motion characteristics. By analyzing the appearance matching degree and the consistency of the motion characteristics, it is beneficial to improve the detection accuracy and matching success rate of obstacles when there are multi-obstacle associated interferences. According to the occlusion degree of the obstacles, determine the tracking accuracy between obstacles in adjacent frame video images, and perform obstacle position perception based on the tracking accuracy. Performing obstacle position perception based on the tracking accuracy determined by the occlusion degree is beneficial to localize and perceive obstacles when the obstacles are occluded, realizing effective tracking of obstacles, and providing reliable support for the path planning and safe driving of driverless vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 The structural block diagram of a road obstacle perception system for a driverless vehicle based on video surveillance provided by an embodiment of the present invention;

[0050] Figure 2 The comparison diagram of video images and radar data provided by an embodiment of the present invention. Among them, (a) is a frame of video image obtained at a certain time, and (b) is a schematic diagram of radar data obtained at the same time;

[0051] Figure 3 The schematic diagram of the bounding boxes of each obstacle provided by an embodiment of the present invention;

[0052] Figure 4 The comparison diagram of the edges obtained by the bounding box and the contour detection provided by an embodiment of the present invention. Among them, (a) is a schematic diagram of the bounding box of a certain obstacle, and (b) is a schematic diagram of the edges obtained by performing contour detection on the obstacle within the bounding box of the obstacle;

[0053] Figure 5 The schematic diagram of the corresponding position fitting line of a certain obstacle provided by an embodiment of the present invention. Detailed implementation manners

[0054] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following will, in conjunction with the accompanying drawings and preferred embodiments, describe in detail a road obstacle perception system for driverless vehicles based on video surveillance according to the present invention, including its specific implementation manners, structures, features, and effects. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0056] It should be noted that the "exemplary" in the embodiments of the present application refers to examples listed for convenience of description, and in some other embodiments, it is not limited to the listed examples.

[0057] It should be noted that to ensure the significance of the calculation results, in the fractional operations in the embodiments of the present invention, when encountering the situation where the denominator is 0, a tuning factor greater than 0 needs to be added to the denominator to prevent the denominator from being 0. The value of the tuning factor is set by the implementer according to the actual situation, and no special restrictions are made in this application.

[0058] The following will specifically describe the specific solution of a road obstacle perception system for driverless vehicles based on video surveillance provided by the present invention with reference to the accompanying drawings.

[0059] Please refer to Figure 1 , which shows a structural block diagram of a road obstacle perception system for driverless vehicles based on video surveillance provided by an embodiment of the present invention. The road obstacle perception system for driverless vehicles based on video surveillance can at least include:

[0060] An acquisition module, configured to acquire road videos through a camera of the driverless vehicle and acquire obstacle distance information around the driverless vehicle through a lidar;

[0061] A first determination module, configured to continuously track and detect the road video through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images, and analyze the motion law according to the obstacle distance information to determine the motion feature consistency of obstacles in adjacent frame video images;

[0062] A second determination module, configured to determine the occlusion degree of the obstacle according to the appearance matching degree and the motion feature consistency when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process;

[0063] A perception module, configured to determine the tracking accuracy between obstacles in adjacent frame video images according to the occlusion degree of the obstacles, and perform position perception of the obstacles according to the tracking accuracy.

[0064] The technical solution of the embodiment of the present application obtains road videos through the camera of the driverless vehicle and obtains the distance information of obstacles around the driverless vehicle through lidar. The road videos are continuously tracked and detected through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images, and the motion law analysis is performed according to the obstacle distance information to determine the consistency of the motion characteristics of obstacles in adjacent frame video images. When the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process, the occlusion degree of the obstacles is determined according to the appearance matching degree and the consistency of the motion characteristics. By analyzing the appearance matching degree and the consistency of the motion characteristics, it is beneficial to improve the detection accuracy and matching success rate of obstacles when there are multi-obstacle associated interferences. According to the occlusion degree of the obstacles, the tracking accuracy between obstacles in adjacent frame video images is determined, and according to the tracking accuracy, the position perception of the obstacles is performed. Performing the position perception of the obstacles based on the tracking accuracy determined by the occlusion degree is beneficial to perform positioning perception of the obstacles when the obstacles are occluded, realize the effective tracking of the obstacles, and provide reliable support for the path planning and safe driving of the driverless vehicle.

[0065] In one implementation manner, a camera may be installed on the top inside of the driverless vehicle (hereinafter referred to as the vehicle). The road videos including surrounding roads, obstacles, traffic signs, etc. are obtained in real time through the camera of the driverless vehicle, and the three-dimensional space information where the vehicle is located is mapped into a two-dimensional image (i.e., several frame video images included in the video). In addition, a lidar is installed on the top outside of the vehicle at a corresponding position consistent with the camera. The lidar uses laser emission and reception of reflected signals to obtain radar data to measure the distance information of surrounding obstacles, so the distance information of obstacles around the vehicle can be obtained.

[0066] Optionally, after obtaining the road videos and the obstacle distance information, preprocessing may be performed to preprocess the collected road videos and obstacle distance information to reduce noise and enhance the data quality. For example, the road videos are processed through a denoising filtering algorithm (such as Gaussian filtering) to remove the noise caused by light changes or motion blur, and the radar data is processed through Kalman filtering to remove invalid points and reduce the noise generated by external interferences. Then, data synchronization and fusion can be performed: synchronize the road videos captured by the camera and the radar data through timestamps to ensure the temporal consistency of the two data, so that each frame video image has corresponding radar data, that is, each frame video image has corresponding obstacle distance information. As Figure 2As shown Figure 2 is a comparison chart of video images and radar data provided by an embodiment of the present invention. Among them, (a) is a frame of video image obtained at a certain time, and (b) is the radar data obtained at the same time.

[0067] In one implementation manner, the first determination module includes a first unit, a second unit, a third unit, and a fourth unit, and is used to continuously track and detect the road video through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images. Specifically:

[0068] The first unit is used to continuously track and detect the road video through an object tracking algorithm to determine the edges of each obstacle in each frame of video image.

[0069] In one implementation manner, an object tracking algorithm such as (Tracktor algorithm) can be used to continuously track and detect the road video. The Tracktor algorithm is an object tracking algorithm based on object detection, which combines object detection and tracking and performs real-time object tracking through continuous detection results. When the Tracktor algorithm continuously tracks and detects the road video, it performs object detection on each frame of video image in the road video to generate the bounding boxes of each obstacle in each frame of video image, such as Figure 3 As shown, since obstacles (such as vehicles, pedestrians, traffic signs, etc.) in the road environment have different shape characteristics, an edge detection algorithm can be used to extract the contour information of the obstacles within the bounding box, so as to obtain the edges of the obstacles, and finally obtain the edges of each obstacle in each frame of video image (each obstacle can have multiple edges), such as Figure 4 As shown Figure 4 is a comparison chart of the edges obtained by detecting the bounding box and the contour provided by an embodiment of the present invention. Among them, (a) is the bounding box of a certain obstacle, and (b) is the edge obtained by detecting the contour of the obstacle within the bounding box of a certain obstacle. Among them, the Tracktor algorithm will match the obstacles in adjacent frame video images, such as calculating the position overlap degree of the bounding boxes for matching, and maintaining and updating the positions of the obstacles through data association and motion models to achieve continuous tracking of each obstacle. The Tracktor algorithm is an existing method, and its principle of continuous tracking of obstacles will not be elaborated too much.

[0070] It should be noted that when calculating the position overlap degree of the bounding boxes for matching, if a match can be achieved, it indicates that the same obstacle has been successfully found in adjacent frame video images. Due to the dynamic and complex characteristics of traffic roads, there are many obstacles and dynamic movements, which may lead to situations such as multi-obstacle association interference and occlusion, resulting in a matching failure. That is, in adjacent frame video images, the obstacle in the previous frame video image cannot be found in the next frame video image. Therefore, the existing method that only relies on the association of bounding boxes is prone to matching errors and tracking failures. Therefore, the method of this application further improves the original object tracking algorithm by introducing dimensions such as appearance matching degree, motion feature consistency, and occlusion degree of obstacles, so as to improve the position perception of obstacles as much as possible and enhance the success rate and accuracy of tracking.

[0071] A second unit, configured to determine the consistency of the change in the edge degree between the obstacles in adjacent frame video images according to the edges of the obstacles in each frame of video image, and determine the geometric feature consistency between the obstacles in adjacent frame video images according to the consistency of the change in the edge degree.

[0072] In one implementation, the consistency of the change in the edge degree between the obstacles in adjacent frame video images is determined according to the edges of the obstacles in each frame of video image , represents Dynamic Time Warping, which is dynamic time warping, is the th obstacle in the th frame of video image at the th edge's chain code sequence (the chain code sequence is obtained by existing methods), is the th frame of video image and the th edge's chain code sequence of any obstacle in the adjacent previous frame of video image, is the th obstacle in the th frame of video image and the change consistency of the edges from long to short corresponding to any obstacle in the adjacent previous frame of video image (i.e., the consistency of the change in the edge degree, when any obstacle corresponds to different obstacles, it can represent the consistency of the change in the edge degree between each obstacle), The smaller it is, the more consistent the edge performance is.

[0073] Then, according to the consistency of the change in the edge degree, the geometric feature consistency between the obstacles in adjacent frame video images is determined :

[0074]

[0075] Among them, represents the The geometric feature consistency of any obstacle in the frame of video image and its adjacent previous frame of video image (it can be understood that when any obstacle corresponds to different obstacles, at this time it can represent the geometric feature consistency of the th obstacle in the frame of video image and its adjacent previous frame of video image for each obstacle, that is, the geometric feature consistency between each obstacle in adjacent frames of video images), represents the number of edges of the obstacle, represents the natural exponential function.

[0076] A third unit, configured to determine the color feature consistency between each obstacle in adjacent frames of video images according to the edges of each obstacle in each frame of video image.

[0077] It should be noted that in a road environment, usually the colors of different obstacles are significantly different, and the colors of vehicles, pedestrians, and road signs are often relatively unique. For example, the body of a car usually has a fixed color (such as red, black, white), while pedestrians may wear clothing of different colors. Therefore, analyzing the color feature consistency is beneficial to the matching and recognition of obstacles.

[0078] First, determine the first frame of video image (such as the th frame of video image) from several frames of video images, and determine the second frame of video image before the first frame of video image. For example, the previous frame of the th frame of video image is the second frame of video image.

[0079] Secondly, respectively determine the first quantity of all pixel points within the bounding box corresponding to the edge of each obstacle in the first frame of video image (that is, the number of pixel points within the bounding box corresponding to the edge of the th obstacle in the th frame of video image) and the second quantity corresponding to pixel points of different colors (that is, the number of pixel points of the th color corresponding to the edge of the th obstacle in the th frame of video image within the bounding box corresponding to the edge), and determine the third quantity corresponding to pixel points of different colors within the bounding box corresponding to the edge of each obstacle in the second frame of video image (that is, the number of pixel points of the th color corresponding to the edge of any obstacle in the previous frame of video image of the th frame of video image. When any obstacle corresponds to different obstacles, it represents each obstacle).

[0080] Furthermore, determine the number of color types contained in the obstacle respectively , and according to the first quantity , the second quantity , the third quantity and the number of color types , determine the color feature consistency of each obstacle in the first frame of video image and each obstacle in the second frame of video image , and the formula is:

[0081]

[0082] wherein, represents the color feature consistency of any obstacle in the th obstacle in the th frame of video image and its adjacent previous frame of video image (similarly, when any obstacle corresponds to different obstacles, it represents each obstacle, that is, the color feature consistency of each obstacle in the first frame of video image and each obstacle in the second frame of video image can be determined). Then, return to the step of determining the first frame of video image from several frames of video images until the color feature consistency between each obstacle in adjacent frames of video images is determined , that is when it is a different value, it represents the color feature consistency between each obstacle in different adjacent frames of video images. Among them, represents the difference in the proportion of pixel points of the same color type of each obstacle in the th obstacle in the th frame of video image and its adjacent previous frame of video image within the bounding box. The smaller this formula is, the more consistent the color proportion is

[0083] The fourth unit is used to determine the appearance matching degree between each obstacle in adjacent frames of video images according to the first product of the geometric feature consistency and the color feature consistency:

[0084] Optionally, in the road environment of an autonomous vehicle, according to the geometric feature consistency and the color feature consistency obtained from video monitoring, the movement trajectory of the same obstacle can be tracked, and the appearance matching degree of the obstacle corresponding to the bounding box in adjacent video frames can be comprehensively obtained. Specifically: according to the geometric feature consistency and the color feature consistency of the first product, determine the appearance matching degree between each obstacle in adjacent frames of video images :

[0085]

[0086] wherein, is the The appearance matching degree of an obstacle in the frame of video image and any obstacle in its adjacent previous frame of video image.

[0087] It should be noted that by introducing the appearance matching degree in the object tracking algorithm, preliminary obstacle matching can be performed. However, due to the complex and dynamic road environment, when there are different obstacles blocking or obstacles with similar appearances (such as cars of the same model and color), matching errors are likely to occur. Therefore, in the embodiments of the present application, the motion law is further analyzed based on the obstacle distance information, the positions of obstacles in future frames are predicted, and whether occlusion occurs is determined, and the obstacle matching errors caused by occlusion are identified and corrected to ensure continuous tracking of obstacles.

[0088] In one implementation manner, the first determination module further includes a fifth unit and a sixth unit, which are used to analyze the motion law according to the obstacle distance information and determine the consistency of the motion characteristics of each obstacle in adjacent frames of video images. Specifically:

[0089] The fifth unit is used to determine the motion trend of each obstacle in each frame of video image according to the obstacle distance information corresponding to each frame of video image.

[0090] First, determine the third frame of video image from several frames of video images. Similarly, taking the frame of video image as an example, assuming that the preset adjacent range is the previous 5 adjacent frames, at this time, several consecutive fourth frame video images within the preset adjacent range of the third frame of video image can be determined, that is, determine the previous 5 consecutive fourth frame video images of the frame of video image, and generate a position fitting line according to the obstacle distance information corresponding to several consecutive fourth frame video images. Because based on the obstacle distance information corresponding to different frames of video images, the positions of each obstacle in each frame of video image can be determined respectively, and thus a corresponding position fitting line can be generated based on the change of positions. As Figure 5 shown, it is the position fitting line corresponding to a certain obstacle.

[0091] Secondly, according to the obstacle distance information corresponding to each frame of video image, determine the first position of each obstacle in any fourth frame of video image and the second position of the corresponding matching obstacle in the previous fourth frame of video image of this fourth frame of video image respectively, and determine the included angle between the line connecting the first position and the second position and the corresponding position fitting line , that is, the th obstacle in the th frame of video image (the third frame of video image) in the adjacent th fourth frame of video image, the first position where any obstacle is located and its previous frame (that is, the - The angle between the line connecting the second position where the obstacle is located and the position fitting line in the fourth frame video image.

[0092] Then, determine the velocity variances of each obstacle in the third frame video image and each obstacle in each fourth frame video image respectively , representing the th obstacle in the th frame video image (the third frame video image) and the velocity variances of each obstacle in each adjacent fourth frame video image. Specifically, the velocity can be obtained by the ratio of the position distance to the time interval, and then the velocity variance can be calculated, which represents the continuity of the movement change of the obstacle. The smaller this value, the more continuous the movement and the more consistent the movement trend.

[0093] Finally, according to the angle and the number of angles , for a preset adjacent range such as 5, perform summation and averaging to obtain the average value of the angles , and respectively determine the movement trend of each obstacle in the third frame video image according to the second product of the average value of the angles and the velocity variance of each obstacle and the natural exponential function : :

[0094]

[0095] Among them, represents the movement trend of the th obstacle in the th frame video image (the third frame video image). Return the steps to determine the third frame video image from several frame video images until the movement trend of each obstacle in each frame video image is determined , that is when taking different values, representing the movement trend of each obstacle in each frame video image. Among them, represents the consistency of the position change direction of the th obstacle in the th frame video image and the same obstacle in its adjacent frame video images. When this value is smaller, it indicates that the movement direction of the obstacle in consecutive frames is more consistent.

[0096] The sixth unit is used to determine the consistency of the movement characteristics of each obstacle in adjacent frame video images according to the movement trend of each obstacle in each frame video image.

[0097] It should be noted that in the road environment of driverless vehicles, there may be significant differences in the movement of long-time objects. However, due to traffic rules, the movement patterns of objects are consistent in a short period. During the process of matching obstacles in the current frame of video image, it is necessary to be consistent with the movement trend of the possible same obstacle in the previous frame. According to the movement trend of obstacles in adjacent frames, the consistency of the movement characteristics of obstacles is obtained.

[0098] First, according to the movement trend of each obstacle in each frame of video image , determine the movement trend difference of each obstacle in adjacent frames of video image , is the movement trend of any obstacle in the adjacent previous frame of video image of the th frame of video image. Similarly, when any obstacle corresponds to different obstacles, it corresponds to the movement trends of each obstacle in the adjacent previous frame of video image of the th frame of video image.

[0099] Secondly, respectively determine the movement characteristic consistency of each obstacle in adjacent frames of video image according to the opposite number of the movement trend difference and the natural exponential function:

[0100]

[0101] where represents the movement characteristic consistency of the th obstacle in the th frame of video image and any obstacle in its adjacent previous frame of video image. Similarly, when any obstacle corresponds to different obstacles, the movement characteristic consistency of each obstacle in adjacent frames of video image is obtained. reflects the difference in the movement trend of the th obstacle in the th frame of video image and any obstacle in its adjacent previous frame of video image. The smaller this formula is, the more consistent the movement trend is, and the more likely it is to be the same obstacle.

[0102] It should be noted that in a complex traffic road environment, there are multiple obstacles, and the vehicles block each other. If there is no obstacle with a high degree of identity in the current frame of video image for the obstacle in the previous frame of video image, it means that the obstacle may be blocked. In adjacent frames of video image, it appears different in appearance, but the movement trajectory of the position information may belong to the same object. Therefore, the occlusion degree of the obstacle can be obtained: if there is an obstacle matching failure in the previous video frame of the current frame, use the uniformly accelerated motion model to predict the position of the obstacle in the previous frame in the current frame, and obtain the occlusion degree through the target information at this position.

[0103] In one embodiment, the second determination module includes a seventh unit, an eighth unit, a ninth unit, and a tenth unit, and is configured to determine the occlusion degree of an obstacle according to the appearance matching degree and the motion feature consistency when the obstacles in adjacent frame video images cannot be correspondingly matched during continuous tracking detection. Specifically:

[0104] The seventh unit is configured to, when the obstacles in adjacent frame video images cannot be correspondingly matched during continuous tracking detection, that is, when the Tracktor algorithm cannot correspondingly match the obstacles in adjacent frame video images during continuous tracking detection of the obstacles, determine a fifth frame video image from several frame video images. Similarly, taking the fifth frame video image as an example, and determine a sixth frame video image which is the previous frame of the fifth frame video image, that is, the previous frame of the

[0105] The eighth unit is configured to predict the positions of the obstacles in the sixth frame video image in the fifth frame video image through a uniformly accelerated motion model to obtain several predicted positions.

[0106] The ninth unit is configured to determine the average appearance matching degree corresponding to each obstacle according to the appearance matching degree between the obstacles in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image (that is, the average appearance matching degree corresponding to each obstacle in the nth obstacle in the adjacent consecutive video frames (the adjacent previous frame video image) of the nth obstacle in the frame video image), and respectively determine the appearance matching degree between the obstacles in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image .

[0107] The tenth unit is configured to respectively determine the occlusion degree of the obstacles at the predicted positions in the fifth frame video image according to the third product of the motion feature consistency and the first difference value of the obstacles in the adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image:

[0108]

[0109] Wherein, represents at a certain predicted position in the th Occlusion degree of an obstacle; The larger it is, the more likely it is that obstacles with the same motion trend are occluded in the current frame.

[0110] Then, return the step of determining the fifth video image from several video images when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking detection until the occlusion degree of the obstacles at each predicted position of each frame of video image is determined. , that is Different values of can correspondingly determine the occlusion degree of the obstacles at each predicted position in different frames of video images.

[0111] In one implementation, the perception module includes a first processing unit, a second processing unit, and a third processing unit, and is used to determine the tracking accuracy between obstacles in adjacent frame video images according to the occlusion degree of the obstacles. Specifically:

[0112] The first processing unit is used to determine the identity between obstacles in adjacent frame video images according to the fourth product of the motion feature consistency and the appearance matching degree.

[0113] It should be noted that in a road environment, due to the complex situation of multiple obstacles, there may be tracking interference between obstacles. According to the appearance matching degree of the obstacles in adjacent frame video images and the motion law of the obstacles in continuous video frames, the identity between obstacles in adjacent frame video images can be obtained:

[0114]

[0115] Among them, represents the identity of the th obstacle in the th frame of video image and any obstacle in the previous adjacent video frame. Similarly, when any obstacle corresponds to different obstacles, the identity between obstacles in adjacent frame video images is obtained.

[0116] The second processing unit is used to determine the corrected matching degree between the obstacles at each predicted position of each frame of video image and the obstacles in the previous frame of video image of this frame according to the identity between obstacles in adjacent frame video images and the occlusion degree of the obstacles.

[0117] It should be noted that when an obstacle is occluded, its matching degree is low. According to the occlusion degree obtained from the motion law of the obstacle, the matching error of the obstacle caused by occlusion is corrected to obtain the corrected matching degree at the position of the obstacle.

[0118] First, assume that the preset value is 1, and respectively determine the preset value 1 and the occlusion degree of the obstacles at each predicted position of each frame of video image. Sum value 。

[0119] Secondly, respectively according to the sum value and the identity between each obstacle in adjacent frame video images of the fifth product, determine the corrected matching degree between the obstacle at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image:

[0120]

[0121] Among them, represents the corrected matching degree between the th obstacle at the predicted position in the frame video image and any obstacle in the previous adjacent frame video image. When any corresponding value takes different values, the corrected matching degree between the obstacle at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image can be obtained 。

[0122] It should be noted that the Tracktor algorithm is used to continuously track the obstacles, and the matching degree of the obstacles is continuously corrected according to their movement trajectories to ensure stable and accurate tracking of multiple obstacles in a complex and dynamic road environment. According to the corrected matching degree at the obstacle position obtained by object detection, the tracking accuracy of the obstacles can be obtained.

[0123] The third processing unit is used to determine the tracking accuracy between each obstacle in adjacent frame video images according to the corrected matching degree.

[0124] First, according to the corrected matching degree between the obstacle at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image , respectively determine the mean value of the corrected matching degrees corresponding to each obstacle , that is Obtained by taking the average.

[0125] Secondly, respectively determine the corrected matching degree between the obstacle at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image and the mean value of the corrected matching degrees corresponding to each obstacle of the second difference value . When this formula is smaller, it indicates that the possibility that the obstacle and the obstacle in the previous consecutive frames have the same movement trajectory (movement trend) is higher, and the tracking accuracy is greater.

[0126] Then, respectively according to the opposite number of the second difference value and the natural exponential function , determine the calculation result , and respectively determine the tracking accuracy between obstacles in adjacent frame video images according to the calculation result and the sixth product of the mean value of the correction matching degrees corresponding to each obstacle. The specific formula is as follows:

[0127]

[0128] wherein, represents the tracking accuracy of the th obstacle in the th frame video image and any obstacle in the immediately preceding frame video image adjacent to it. takes different values corresponding to each obstacle, so as to determine the tracking accuracy between obstacles in adjacent frame video images.

[0129] In one implementation manner, according to the tracking accuracy, the position perception of the obstacle is performed. Specifically: a tracking threshold such as 0.7 can be set in advance. If the tracking accuracy of a certain obstacle is greater than 0.7, it indicates that the possibility that a certain obstacle in consecutive frame video images is the motion trajectory (motion trend) of the same obstacle is high. Then it is determined that the obstacle can be accurately detected in the current frame video image. And if the obstacle in the previous frame video image is successfully matched, at this time the tracking is successful, and the obstacle distance information of the obstacle is updated.

[0130] Optionally, when performing the position perception of the obstacle, according to the obtained motion law of the obstacle, the uniform acceleration motion model is used to predict the future motion trajectory of the obstacle, judge the position change of the obstacle in advance, and provide a basis for path planning. And when the obstacle may be occluded, the occlusion degree and tracking accuracy of the obstacle are determined by the above method for occlusion detection and repair to improve the tracking accuracy; moreover, when the obstacle reappears in the field of view after being occluded, the obstacle is re-identified and matched. It can be judged whether it is the obstacle tracked before by comparing the appearance matching degree or the consistency of motion characteristics of the obstacle, so as to avoid incorrect trajectory matching. Finally, on the basis of ensuring the normal matching of the obstacle, the obstacle can be classified and dynamically characterized to identify the type of the obstacle (such as pedestrian, vehicle, static object, etc.) and motion characteristics (such as speed, acceleration, etc.). The unmanned vehicle system can then perform real-time path planning to adjust the driving route of the vehicle, ensure that the vehicle finds the optimal driving path in the dynamic environment, and safely bypass the obstacle.

[0131] The method according to the embodiment of the present application collects the road video of the driverless vehicle, analyzes the movement law through the obstacle distance information of the obstacles in the consecutive frame video images to distinguish different obstacles, prevents multi-obstacle interference in tracking, extracts the consistency of the movement characteristics of the obstacles in the consecutive video frames, and combines the spatial position relationship determined by the obstacle distance information relative to the driverless vehicle to achieve continuous tracking of the obstacles, improving the perception ability of the driverless vehicle in a complex and dynamic road environment. At the same time, during the process of tracking the obstacles, the obstacles in the frame video images are identified and located. By associative matching of the obstacles in the consecutive frame video images, the same obstacle in different frame video images is judged, and cross-frame continuous tracking is achieved, avoiding the limitations of the static obstacle detection method, and being able to stably track the obstacles (such as pedestrians, other vehicles, etc.) in the dynamic scene, ensuring continuous recognition of the obstacles. In addition, to prevent tracking failure of the obstacles caused by occlusion, analyzing the movement law of the obstacles to determine the consistency of the movement characteristics can improve the positioning accuracy of the obstacles and predict the behavior trend of the obstacles. Finally, by identifying and predicting the occlusion situation, the correction matching degree of the continuous obstacles is corrected. Even if the obstacle disappears briefly, its position can be restored through the trajectory, ensuring the stability and robustness of the obstacle perception system in a complex environment. Especially in a high-density traffic or rapidly changing scene, it avoids misjudgment caused by obstacle occlusion, enables the road obstacle perception system of the driverless vehicle to adapt to environmental changes in real time, maintains efficient and accurate obstacle tracking, and thus provides strong support for the safe driving, path planning and decision-making of the driverless vehicle.

[0132] It should be noted that the above sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or consecutive order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0133] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key point of each embodiment is to illustrate the differences from other embodiments.

Claims

1. A road obstacle perception system for driverless vehicles based on video surveillance, characterized in that, The system includes: An acquisition module, configured to acquire road videos through a camera of an autonomous vehicle and acquire obstacle distance information around the autonomous vehicle through a lidar; A first determination module, configured to continuously track and detect the road videos through an object tracking algorithm to determine the appearance matching degree between obstacles in adjacent frame video images, and perform motion law analysis according to the obstacle distance information to determine the motion feature consistency of obstacles in adjacent frame video images; A second determination module, configured to determine the occlusion degree of an obstacle according to the appearance matching degree and the motion feature consistency when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process; The second determination module includes a seventh unit, an eighth unit, a ninth unit, and a tenth unit: The seventh unit, configured to determine a fifth frame video image from several frame video images and determine a sixth frame video image which is the previous frame of the fifth frame video image when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process; The eighth unit, configured to predict the positions of obstacles in the sixth frame video image in the fifth frame video image through a uniformly accelerated motion model to obtain several predicted positions; The ninth unit, configured to determine the average value of the appearance matching degree corresponding to each obstacle according to the appearance matching degree between obstacles in adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image, and respectively determine the first difference value between the appearance matching degree between obstacles in adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image and the average value of the appearance matching degree corresponding to each obstacle; The tenth unit, configured to respectively determine the occlusion degree of obstacles at the predicted positions in the fifth frame video image according to the third product of the motion feature consistency and the first difference value between obstacles in adjacent frame video images corresponding to the fifth frame video image and the sixth frame video image, and return the step of determining a fifth frame video image from several frame video images when the obstacles in adjacent frame video images cannot be correspondingly matched during the continuous tracking and detection process until the occlusion degree of obstacles at the predicted positions in each frame video image is determined; A perception module, configured to determine the tracking accuracy between obstacles in adjacent frame video images according to the occlusion degree of obstacles, and perform position perception of obstacles according to the tracking accuracy; wherein, when an obstacle is occluded, the occlusion degree obtained according to the motion law of the obstacle is used to correct the obstacle matching error caused by occlusion to obtain the corrected matching degree at the position of the obstacle; The perception module includes a first processing unit, a second processing unit, and a third processing unit: The first processing unit, configured to determine the identity between obstacles in adjacent frame video images according to the fourth product of the motion feature consistency and the appearance matching degree; The second processing unit is configured to determine a corrected matching degree between the obstacles at each predicted position in each frame of video image and the obstacles in the previous frame of video image of this frame of video image according to the identity between the obstacles in the adjacent frame video images and the occlusion degree of the obstacles. The third processing unit is configured to determine the tracking accuracy between the obstacles in the adjacent frame video images according to the corrected matching degree.

2. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 1, wherein: The first determining module includes a first unit, a second unit, a third unit, and a fourth unit: The first unit is configured to continuously track and detect the road video through an object tracking algorithm to determine the edges of the obstacles in each frame of video image. The second unit is configured to determine the consistency of the edge degree change between the obstacles in the adjacent frame video images according to the edges of the obstacles in each frame of video image, and determine the geometric feature consistency between the obstacles in the adjacent frame video images according to the consistency of the edge degree change. The third unit is configured to determine the color feature consistency between the obstacles in the adjacent frame video images according to the edges of the obstacles in each frame of video image. The fourth unit is configured to determine the appearance matching degree between the obstacles in the adjacent frame video images according to the first product of the geometric feature consistency and the color feature consistency.

3. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 2, characterized in that: The determining the color feature consistency between the obstacles in the adjacent frame video images according to the edges of the obstacles in each frame of video image includes: Determining a first frame of video image from a plurality of frames of video images, and determining a second frame of video image that is the previous frame of the first frame of video image. Respectively determining a first quantity of all pixel points and a second quantity of pixel points corresponding to different colors within the bounding box corresponding to the edge of each obstacle in the first frame of video image, and determining a third quantity of pixel points corresponding to different colors within the bounding box corresponding to the edge of each obstacle in the second frame of video image. Respectively determining the number of color types included in the obstacles, and determining the color feature consistency between the obstacles in the first frame of video image and the obstacles in the second frame of video image according to the first quantity, the second quantity, the third quantity, and the number of color types, and returning to the step of determining the first frame of video image from the plurality of frames of video images until the color feature consistency between the obstacles in the adjacent frame video images is determined.

4. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 2 or 3, characterized in that: The first determining module further includes a fifth unit and a sixth unit: The fifth unit is configured to determine the motion trend of each obstacle in each frame of video image according to the obstacle distance information corresponding to each frame of video image. The sixth unit is configured to determine the motion feature consistency between the obstacles in the adjacent frame video images according to the motion trend of each obstacle in each frame of video image.

5. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 4, wherein: The determining the motion trend of each obstacle in each frame of video image according to the obstacle distance information corresponding to each frame of video image includes: Determine the third video image from a number of video images, determine a number of consecutive fourth video images within a preset proximity range of the third video image, and generate a position fitting line according to the obstacle distance information corresponding to the number of consecutive fourth video images; According to the obstacle distance information corresponding to each video image, respectively determine the first position of each obstacle in any one of the fourth video images and the second position of the corresponding matching obstacle in the previous fourth video image of the fourth video image, and respectively determine the angle between the straight line connecting the first position and the second position and the corresponding position fitting line; Respectively determine the velocity variances of each obstacle in the third video image and each obstacle in each of the fourth video images; Sum and average the angles according to the angle and the number of the angles to obtain an average angle value, and respectively determine the motion trends of each obstacle in the third video image according to the second product of the average angle value and the velocity variances of each obstacle and the natural exponential function, and return to the step of determining the third video image from a number of video images until the motion trends of each obstacle in each video image are determined.

6. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 4, characterized in that: The determination of the motion feature consistency of each obstacle in adjacent video images according to the motion trends of each obstacle in each video image includes: Determine the motion trend differences of each obstacle in adjacent video images according to the motion trends of each obstacle in each video image; Respectively determine the motion feature consistency of each obstacle in adjacent video images according to the opposite number of the motion trend differences and the natural exponential function.

7. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 1, wherein: The determination of the corrected matching degrees of each obstacle at each predicted position in each video image and each obstacle in the previous frame video image of the video image according to the identity between each obstacle in the adjacent video images and the occlusion degree of the obstacle includes: Respectively determine the sum values of a preset value and the occlusion degrees of each obstacle at each predicted position in each video image; Respectively determine the corrected matching degrees of each obstacle at each predicted position in each video image and each obstacle in the previous frame video image of the video image according to the fifth product of the sum values and the identity between each obstacle in the adjacent video images.

8. The road obstacle perception system for driverless vehicles based on video surveillance according to claim 7, characterized in that: The determination of the tracking accuracy between each obstacle in adjacent video images according to the corrected matching degrees includes: Respectively determine the average values of the corrected matching degrees corresponding to each obstacle according to the corrected matching degrees of each obstacle at each predicted position in each video image and each obstacle in the previous frame video image of the video image; Respectively determine the second difference values between the corrected matching degrees of each obstacle at each predicted position in each video image and each obstacle in the previous frame video image of the video image and the average values of the corrected matching degrees corresponding to each obstacle; Determine the calculation results according to the opposite number of the second difference value and the natural exponential function respectively, and determine the tracking accuracy between each obstacle in adjacent frame video images according to the calculation results and the sixth product of the mean value of the correction matching degrees corresponding to each obstacle respectively.

Images