Autonomous driving water surface scene perception method, device, equipment and storage medium
By using binocular cameras and multi-objective tracking networks in the autonomous driving system, combined with the semi-global matching SGM algorithm, the problem of limited field angles of a single camera and inconsistent coordinate system in the water surface scene is solved, and the panoramic perception and coordinate system are realized , improves the safety and perceived accuracy of autonomous driving surface scenarios.
Patent Information
- Application Number
- CN202210616024.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-05-31
AI Technical Summary
In the water surface scene, the existing autonomous driving system has a limited field of view angle of a single camera and inconsistent scene data coordinate system, resulting in low perceptual accuracy, easy collision with obstacles, risk of accidents, and affect safety.
Using a vehicle gimbal equipped with a binocular camera, the multi-target tracking network is trained by obtaining historical video data, and the real-time video data is used to track obstacles and segment instances. The semi-global matching SGM algorithm is used to convert three-dimensional coordinates to realize panoramic perception and coordinate system one. Drive the gimbal to focus on key points.
The panoramic perception of a single binocular camera is realized, reducing the probability of collision obstacles in surface scenes of autonomous driving, avoiding the risk of accidents, and improving the accuracy and safety of autonomous driving surface scene perception.
Smart Images

Figure CN114972996B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving water surface scene perception method, device, equipment and storage medium. Background Art
[0002] The scene perception module is an important module of the autonomous driving system, which completes the task of real-time perception and understanding of the environment around the vehicle; scene perception is crucial to the behavioral decision-making and motion control of autonomous driving, providing reliable scene data for these operations.
[0003] Currently, scene perception generally uses a combination of high-tech sensors and cameras, combined with advanced software technology, to perceive and understand the environment in real time. However, the field of view of a single camera is limited and can only perceive scenes within a range of about 90°. One solution is to configure multiple cameras to perceive 360° panoramic scenes, but this is expensive and requires higher computing performance to process more image data in real time. In addition, in the process of expanding autonomous driving technology from land to water, due to interference from factors such as water flow fluctuations, the coordinate system of the perceived scene data will undergo different transformations and be difficult to unify, resulting in autonomous driving in water scenes being prone to collisions with obstacles, posing a risk of accidents and affecting the safety of autonomous driving. Summary of the invention
[0004] The main purpose of the present invention is to provide an autonomous driving water surface scene perception method, device, equipment and storage medium, aiming to solve the technical problems in the prior art that the current single camera perception field of view is limited, the scene data coordinate system is not unified, and the scene perception accuracy is low, which makes the autonomous driving of water surface scenes prone to collision with obstacles, there is a risk of accidents, and affects the safety of autonomous driving.
[0005] In a first aspect, the present invention provides an autonomous driving water surface scene perception method, the autonomous driving water surface scene perception method comprising the following steps:
[0006] Acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire a trained multi-target tracking network;
[0007] Using a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, and performing obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result;
[0008] The three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result are obtained according to the semi-global matching SGM algorithm, and the three-dimensional coordinates are transformed to obtain a point set in the scene coordinate system. The global scene representation is obtained according to the point set to drive the gimbal to focus on the key position.
[0009] Optionally, the acquiring historical video data of the water surface scene, training an initial multi-target tracking network according to the historical video data, and acquiring the trained multi-target tracking network includes:
[0010] Acquire channel videos of water surface scenes at different time periods, manually annotate and mask the channel videos, and obtain historical video data in which channel obstacles are marked with boxes and masked with instances in the channel videos;
[0011] An initial multi-object tracking network with instance segmentation branches added is trained according to the historical video data to obtain a trained multi-object tracking network.
[0012] Optionally, the using a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, performing obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result includes:
[0013] The binocular camera and the inertial measurement unit (IMU) are fixedly mounted on the vehicle gimbal, and the scene coordinate system and the scene pixel coordinate system are initialized;
[0014] Acquiring real-time video data of the water surface scene using the binocular camera;
[0015] Obstacle tracking and instance segmentation are performed on the real-time video data according to the trained multi-target tracking network to obtain segmentation results of real-time obstacle detection at multiple moments in the video data obtained by the left and right cameras respectively.
[0016] Optionally, after performing obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network to obtain segmentation results of real-time obstacle detection at multiple moments in the video data obtained by the left and right cameras respectively, the autonomous driving water surface scene perception method further includes:
[0017] The IDs of the same obstacle in the video data obtained by the left and right cameras are matched by the Hungarian matching algorithm, and the ID of the obstacle in the video data obtained by the right camera is changed to the ID of the obstacle in the video data obtained by the left camera.
[0018] Optionally, obtaining three-dimensional coordinates corresponding to non-background pixels in the segmentation result according to a semi-global matching SGM algorithm, performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation according to the point set, and driving the gimbal to focus on a key position includes:
[0019] Performing stereo matching on pixel data within detection frames of the obstacle in the left and right video frames at the same time in the segmentation result by using a semi-global matching (SGM) algorithm to obtain a disparity map based on the left camera;
[0020] Eliminate the background pixels in the disparity map, and calculate the three-dimensional coordinates of the non-background pixels in the camera coordinate system of the left camera by epipolar geometry;
[0021] The three-dimensional coordinates are transformed to obtain a point set in a scene coordinate system, a global scene representation is obtained according to the point set, and the gimbal is driven to focus on a key position.
[0022] Optionally, the performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation according to the point set, and driving the gimbal to focus on a key position includes:
[0023] Collect the inertial measurement output results of the vehicle gimbal at multiple times;
[0024] Performing coordinate transformation on the three-dimensional coordinates according to the inertial measurement output result to obtain a point set corresponding to the coordinates of the obstacle pixel points in the scene coordinate system;
[0025] A global scene representation is obtained according to the point set, and the gimbal is driven to focus on key locations.
[0026] Optionally, obtaining a global scene representation according to the point set and driving the gimbal to focus on a key position includes:
[0027] Obtaining the obstacle ID corresponding to the point set, and calculating the convex hull according to the obstacle ID;
[0028] Calculate the current speed of each obstacle at the kth moment according to the position change of the centroid of the convex hull, and estimate the predicted centroid and predicted speed of each obstacle at the next moment according to the current speed and Kalman filter;
[0029] When the current system mode is the attention mode, the coordinates of the convex hull are converted from the scene coordinate system to the scene pixel coordinate system to obtain a global scene representation, and the target obstacle with the highest collision risk value is calculated according to the global scene representation;
[0030] The gimbal is driven to turn according to the predicted center of mass, the predicted speed and the target obstacle and focuses on the key position with the highest risk value.
[0031] In a second aspect, in order to achieve the above-mentioned purpose, the present invention further proposes an autonomous driving water surface scene perception device, the autonomous driving water surface scene perception device comprising:
[0032] A data acquisition module, used to acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire a trained multi-target tracking network;
[0033] A segmentation module is used to obtain real-time video data of the water surface scene using a vehicle gimbal equipped with a binocular camera, and perform obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result;
[0034] The scene representation and understanding module is used to obtain the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, perform coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtain a global scene representation based on the point set, and drive the gimbal to focus on the key position.
[0035] In the third aspect, to achieve the above-mentioned purpose, the present invention also proposes an autonomous driving water surface scene perception device, which includes: a memory, a processor, and an autonomous driving water surface scene perception program stored in the memory and executable on the processor, wherein the autonomous driving water surface scene perception program is configured to implement the steps of the autonomous driving water surface scene perception method as described above.
[0036] In a fourth aspect, in order to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which an autonomous driving water surface scene perception program is stored, and when the autonomous driving water surface scene perception program is executed by a processor, the steps of the autonomous driving water surface scene perception method as described above are implemented.
[0037] The automatic driving surface scene perception method proposed in the present invention obtains historical video data of the surface scene, trains an initial multi-target tracking network according to the historical video data, and obtains the trained multi-target tracking network; uses a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the surface scene, and performs obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network to obtain a segmentation result; obtains three-dimensional coordinates corresponding to non-background pixels in the segmentation result according to a semi-global matching SGM algorithm, performs coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtains a global scene representation according to the point set, drives the gimbal to focus on key positions, and can realize panoramic perception of a single binocular camera. The closed-loop system with feedback ensures automatic perception of the perception platform, greatly saves manpower and material costs, maintains the unity of the scene data coordinate system in the surface scene, is conducive to the expansion of the automatic driving technology to the surface field, improves the accuracy of scene perception, reduces the probability of automatic driving of the surface scene colliding with obstacles, avoids accident risks, ensures the safety of automatic driving, and improves the speed and efficiency of automatic driving surface scene perception. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;
[0039] Figure 2 This is a flow chart of the first embodiment of the method for sensing water surface scenes in an autonomous driving system of the present invention;
[0040] Figure 3 This is a flow chart of a second embodiment of the method for sensing a water surface scene in an autonomous driving system according to the present invention;
[0041] Figure 4 This is a flow chart of a third embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention;
[0042] Figure 5 This is a flow chart of a fourth embodiment of the method for sensing a water surface scene in an autonomous driving system according to the present invention;
[0043] Figure 6 This is a flow chart of a fifth embodiment of the method for sensing a water surface scene in an autonomous driving system according to the present invention;
[0044] Figure 7 This is a flow chart of a sixth embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention;
[0045] Figure 8 A schematic diagram of a closed-loop system of the autonomous driving water surface scene perception method of the present invention;
[0046] Fig. 9 This is a functional module diagram of the first embodiment of the autonomous driving water surface scene perception device of the present invention.
[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0049] The solution of the embodiment of the present invention is mainly as follows: by acquiring historical video data of the water surface scene, an initial multi-target tracking network is trained according to the historical video data to obtain the trained multi-target tracking network; using a vehicle gimbal equipped with a binocular camera to acquire real-time video data of the water surface scene, obstacle tracking and instance segmentation are performed on the real-time video data according to the trained multi-target tracking network to obtain a segmentation result; according to a semi-global matching SGM algorithm, the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result are obtained, the three-dimensional coordinates are converted to obtain a point set in the scene coordinate system, a global scene representation is obtained according to the point set, and the gimbal is driven to focus on the key position, so as to realize a single binocular camera. The panoramic perception of the aircraft and the closed-loop system with feedback ensure the automatic perception of the perception platform, which greatly saves manpower and material costs, maintains the unity of the scene data coordinate system in the water surface scene, is conducive to the expansion of autonomous driving technology to the water surface field, improves the accuracy of scene perception, reduces the probability of autonomous driving in water surface scenes colliding with obstacles, avoids accident risks, ensures the safety of autonomous driving, and improves the speed and efficiency of autonomous driving surface scene perception. It solves the technical problems in the existing technology that the current single camera perception field angle is limited, the scene data coordinate system is not unified, and the scene perception accuracy is low, which makes autonomous driving in water surface scenes easy to collide with obstacles, there is an accident risk, and affects the safety of autonomous driving.
[0050] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.
[0051] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (Non-Volatile Memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.
[0052] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0053] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating device, a network communication module, a user interface module, and an automatic driving surface scene perception program.
[0054] The device of the present invention calls the autonomous driving water surface scene perception program stored in the memory 1005 through the processor 1001, and executes the operations in the embodiment of the autonomous driving water surface scene perception method described below.
[0055] Based on the above hardware structure, an embodiment of the autonomous driving water surface scene perception method of the present invention is proposed.
[0056] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the autonomous driving water surface scene perception method of the present invention.
[0057] In a first embodiment, the autonomous driving water surface scene perception method comprises the following steps:
[0058] Step S10: Acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire a trained multi-target tracking network.
[0059] It should be noted that the historical video data are multiple sections of channel video data collected at different times. The preset initial multi-target tracking network (Joint Detection and Embedding, JDE) can be trained through the historical video data to obtain a trained multi-target tracking network.
[0060] Step S20: using a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, and performing obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network to obtain a segmentation result.
[0061] It can be understood that the real-time video data of the water surface scene can be obtained by using a vehicle gimbal equipped with a binocular camera. Generally, a binocular camera fixed on the gimbal can be used to shoot the scene in front of the camera; the real-time video data can be tracked and instance segmented by the trained multi-target tracking network, thereby obtaining the corresponding segmentation results.
[0062] Step S30, obtaining the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, performing coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtaining a global scene representation according to the point set, and driving the gimbal to focus on the key position.
[0063] It should be understood that by obtaining the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result through the semi-global matching (SGM) algorithm, and further performing coordinate transformation on the three-dimensional coordinates, it is possible to obtain a point set in the scene coordinate system, and then obtain a global scene representation based on the point set to drive the gimbal to focus on the key position.
[0064] Through the above scheme, this embodiment obtains historical video data of the water surface scene, trains an initial multi-target tracking network according to the historical video data, and obtains the trained multi-target tracking network; uses a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, and performs obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network to obtain a segmentation result; obtains the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, performs coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtains a global scene representation according to the point set, drives the gimbal to focus on the key position, and can realize panoramic perception of a single binocular camera. The closed-loop system with feedback ensures automatic perception of the perception platform, greatly saves manpower and material costs, maintains the unification of the scene data coordinate system under the water surface scene, is conducive to the expansion of autonomous driving technology to the water surface field, improves the accuracy of scene perception, reduces the probability of autonomous driving in water surface scenes colliding with obstacles, avoids accident risks, ensures the safety of autonomous driving, and improves the speed and efficiency of autonomous driving surface scene perception.
[0065] Further, Figure 3 FIG. 2 is a flow chart of a second embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention. Figure 3 As shown, a second embodiment of the method for sensing a water surface scene in an autonomous driving system of the present invention is proposed based on the first embodiment. In this embodiment, step S10 specifically includes the following steps:
[0066] Step S11, obtaining channel videos of water surface scenes at different time periods, manually marking and instance masking the channel videos, and obtaining historical video data of channel obstacles with marked boxes and instance masking in the channel videos.
[0067] It should be noted that by obtaining waterway videos of water surface scenes at different time periods, the waterway videos can be manually labeled and instance masked to obtain corresponding historical video data; taking inland waterway videos as an example, multiple inland waterway videos at different times can be collected, and manually labeled marking frames and instance masked can be performed to obtain marking frames and instance masks of inland waterway obstacles in the video.
[0068] In the specific implementation, the collected inland waterway video is:
[0069] dataV k (u, v)
[0070] u∈[1,O], v∈[1,P], k∈[1,K]
[0071] Among them, dataV k (u, v) is the pixel at the uth row and vth column of the video frame at the kth moment of the acquired video, O is the number of rows of the video frame, P is the number of columns of the video frame, and K is the number of frames of the video.
[0072] The marking boxes of inland waterway obstacles in the video are:
[0073] bbox id,k =[x,y,w,h]
[0074] Among them, bbox id,k is the marking box of the obstacle with ID id in the video frame at the kth moment of the captured video, x is the horizontal coordinate of the center point of the marking box, y is the vertical coordinate of the center point of the marking box, w is the length of the marking box, and h is the height of the marking box.
[0075] The example mask of the inland waterway obstacle in the video is:
[0076] cell u,v,k =c
[0077] c∈[0,C]
[0078] Among them, cell u,v,k is the pixel in the uth row and vth column of the video frame at the kth moment, c is the category of the pixel, C+1 is the total number of categories, and 0 represents the background.
[0079] In a specific implementation, C+1=4 can be set as the total number of categories, where 0 represents background, 1 represents ship, 2 represents shore and island, and 3 represents others.
[0080] Step S12: training an initial multi-target tracking network with instance segmentation branches added according to the historical video data to obtain a trained multi-target tracking network.
[0081] It can be understood that by training the initial multi-target tracking network with an instance segmentation branch added to the historical video data, a trained multi-target tracking network suitable for waterway surface scenes can be obtained.
[0082] In the specific implementation, the loss function of the JDE network with instance segmentation branch added is:
[0083]
[0084] Among them, L total,kis the total loss of the video frame at the kth moment, M is the number of all prediction heads of the video frame at the kth moment, is the classification loss in the i-th prediction head of the video frame at the k-th moment, is the learnable variance hyperparameter of the loss, is the regression loss in the i-th prediction head of the video frame at the k-th moment, is the learnable variance hyperparameter of the loss, is the appearance embedding loss in the i-th prediction head of the video frame at the k-th time, is the learnable variance hyperparameter of the loss, is the masking loss in the i-th prediction head of the video frame at the k-th moment, is the learnable variance hyperparameter of the loss.
[0085] The classification loss in the i-th prediction head of the video frame at the k-th time is:
[0086]
[0087]
[0088] Among them, m is the index of the anchor point, p m,k is the predicted probability that the anchor point m in the video frame at the kth moment is the target. When the result box corresponding to the anchor point m in the video frame at the kth moment is the target, otherwise, is the logarithmic loss of the target class or the background class.
[0089] The regression loss in the i-th prediction head of the video frame at the k-th time is:
[0090]
[0091]
[0092] t m,k ={t m,x,k , t m,y,k , t m,w,k , t m,h,k}
[0093]
[0094] Among them, t m,k ={t m,x,k , t m,y,k , t m,w,k , t m,h,k} is the vector of parameterized coordinates of the result box corresponding to the anchor point m in the video frame at the kth moment, t m,x,k , t m,y,kare the x and y offsets of the result box corresponding to the anchor point m in the video frame at the kth moment relative to the anchor point m classified as the target, t m,w,k , t m,h,k They are the scale factors of w and h of the result box corresponding to the anchor point m classified as the target in the video frame at the kth moment relative to the anchor point m. is the vector of parameterized coordinates of the marker box corresponding to the anchor point m in the video frame at the kth moment, are the x and y offsets of the marked box corresponding to the anchor point m in the video frame at the kth moment relative to the anchor point m classified as the target, are the scale factors w and h of the marked box corresponding to the anchor point m classified as the target in the video frame at the kth moment relative to the anchor point m. is the regression loss, where R is SmoothL 1 Functions. In particular:
[0095]
[0096]
[0097] x m,k , x′ m,k , are the horizontal coordinates of the center point of the target result box, the center point of the anchor box, and the center point of the marker box corresponding to the anchor point m in the video frame at the kth moment, respectively. m,k , y′ m,k , are the ordinates of the center point of the target result box, the center point of the anchor box, and the center point of the marker box corresponding to the anchor point m in the video frame at the kth moment, respectively. m,k , w′ m,k , are the width of the target result box, the width of the anchor box, and the width of the marker box corresponding to the anchor point m in the video frame at the kth moment, respectively. m,k , h′ m,k , They are respectively the height of the target result box, the height of the anchor box, and the height of the marker box corresponding to the anchor point m in the video frame at the kth moment.
[0098] Accordingly, the specific method for determining the target category or background category is as follows:
[0099] When the marked box in the video frame at time k The target result box [x m,k ,y m,k , w m,k ,h m,k] is greater than α = 0.7, or the intersection and union ratio is the largest among all anchor points contained in the video frame at the kth moment, then it is the target; when the marked box in the video frame at the kth moment The target result box [x m,k ,y m,k , w m,k ,h m,k ] is less than α=0.3, it is the background.
[0100] The appearance embedding loss in the i-th prediction head of the video frame at the k-th time is:
[0101]
[0102]
[0103] in, is the appearance embedding of the result box of the trajectory with ID id in the video frame at the kth moment, called the benchmark sample, is the appearance embedding of the marker box of the track with ID id in the video frame at the kth moment, called a positive sample, The appearance embedding of the marker box of the trajectory whose ID is not id in the video frame at the kth moment is called a negative sample, and a total of N such negative samples are randomly selected. is the cross entropy loss.
[0104] The masking loss in the i-th prediction head of the video frame at the k-th time is:
[0105]
[0106]
[0107] Where a is the width and height of the result frame after deformation in the video frame at the kth moment, and y is u,v,k is the label category of the pixel in the uth row and vth column of the result box after deformation processing in the video frame at the kth moment, is the predicted category of the pixel in the uth row and vth column of the result box after deformation processing in the video frame at the kth moment, is the binary cross entropy loss. In particular, the instance segmentation branch predicts C different categories of masks for each result box. When the result box category is c, only the loss of the mask of category c is calculated.
[0108] The momentum optimization method is used during training. When the loss value generated by the model meets the set requirements or reaches the maximum number of iterations N, the training of the model is terminated and the model parameters are saved.
[0109] Through the above scheme, this embodiment obtains channel videos of water surface scenes at different time periods, manually annotates and masks the channel videos, and obtains historical video data in which channel obstacles are marked with marked boxes and masked by instances in the channel videos; an initial multi-target tracking network with an instance segmentation branch added is trained according to the historical video data to obtain the trained multi-target tracking network, and panoramic perception of a single binocular camera can be achieved through the trained multi-target tracking network, further improving the speed and efficiency of autonomous driving water surface scene perception.
[0110] Further, Figure 4 FIG. 1 is a flow chart of a third embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention. Figure 4 As shown, a third embodiment of the method for sensing a water surface scene in an autonomous driving system of the present invention is proposed based on the first embodiment. In this embodiment, step S20 specifically includes the following steps:
[0111] Step S21: fix the binocular camera and the inertial measurement unit IMU on the vehicle gimbal, and initialize the scene coordinate system and the scene pixel coordinate system.
[0112] It should be noted that the binocular camera and the inertial measurement unit (IMU) are fixedly installed on the gimbal, and the binocular camera and IMU are used to perceive and correct the environmental information in a specific direction. Based on the deep learning model, epipolar geometry and filter, the corrected data is processed to form a scene representation. Then the scene is understood, and the direction that the camera needs to perceive at the next moment is decided. The gimbal is driven to turn to that direction to perceive again to form a closed loop.
[0113] It can be understood that the binocular camera and IMU are fixed on the gimbal, the z-axis of the IMU is aligned with the rotation axis of the gimbal, the y-axis points to the north, and the binocular camera baseline is equally divided and perpendicular to the y-axis of the IMU toward the north; based on the installation position, the relationship and transformation between the gimbal coordinate system, the Local East-North-Up Coordinates (ENU coordinate system) with the IMU as the origin, and the camera coordinate system are obtained.
[0114] It should be understood that the scene coordinate system and the scene pixel coordinate system are initialized, that is, the scene coordinate system is an ENU coordinate system with the IMU position at a certain moment as the origin and the z-axis as 0, the x-axis points to the east, and the y-axis points to the north, and the scene pixel coordinate system is the projection of the scene coordinate system on the two-dimensional image.
[0115] Step S22: using the binocular camera to obtain real-time video data of the water surface scene.
[0116] It can be understood that by using a binocular camera fixed on a gimbal to shoot the scene in front of the camera, the video data of the scene in front can be used as real-time video data of the water surface scene.
[0117] Step S23: performing obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network, and obtaining segmentation results of real-time obstacle detection at multiple moments in the video data obtained by the left and right cameras respectively.
[0118] It should be understood that the model corresponding to the trained multi-target tracking network with added instance segmentation branches performs obstacle tracking and instance segmentation on the video data obtained by the left and right cameras respectively, and obtains real-time detection and segmentation results of obstacles at multiple moments in the video data obtained by the left and right cameras respectively.
[0119] Through the above scheme, this embodiment initializes the scene coordinate system and the scene pixel coordinate system by fixing the binocular camera and the inertial measurement unit IMU on the vehicle gimbal; uses the binocular camera to obtain real-time video data of the water surface scene; performs obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network, and obtains segmentation results of real-time obstacle detection at multiple moments in the video data obtained by the left and right cameras respectively; can realize panoramic perception of a single binocular camera, and a closed-loop system with feedback ensures automatic perception of the perception platform, which greatly saves manpower and material costs.
[0120] Further, Figure 5 FIG. 4 is a flow chart of a fourth embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention. Figure 5 As shown, a fourth embodiment of the automatic driving water surface scene perception method of the present invention is proposed based on the third embodiment. In this embodiment, after the step S23, the automatic driving water surface scene perception method further includes the following steps:
[0121] Step S24: Match the IDs of the same obstacle in the video data obtained by the left and right cameras respectively by using the Hungarian matching algorithm, and change the ID of the obstacle in the video data obtained by the right camera to the ID of the obstacle in the video data obtained by the left camera.
[0122] It should be noted that based on the detection result of the moment when the obstacle first appears in the videos obtained by the left and right cameras, the ID of the same obstacle in the video data obtained by the left and right cameras is matched through the Hungarian matching algorithm, and the ID of the obstacle in the video data obtained by the right camera is changed to the ID of the obstacle in the video data obtained by the left camera, thereby realizing the unification of the ID of the video data and facilitating subsequent data processing.
[0123] In the specific implementation, the real-time detection results of obstacles in the video frames around the kth moment are:
[0124]
[0125]
[0126] in, The IDs of the left video frame at the kth moment are id l The horizontal and vertical coordinates of the center point of the obstacle result box, the width and height of the result box, and the appearance embedding, c u,v,l,k is the predicted category of the pixel in the uth row and vth column of the obstacle result box in the left video frame at the kth moment, The IDs of the right video frame at the kth moment are id r The horizontal and vertical coordinates of the center point of the obstacle result box, the width and height of the result box, and the appearance embedding, c r,k,u,v It is the predicted category of the pixel in the uth row and vth column of the obstacle result box in the right video frame at the kth moment.
[0127] Accordingly, the cost matrix for matching the IDs of the same obstacle in the left and right video frames is:
[0128] C=λA e +(1-λ)A m
[0129] Among them, A e is the appearance embedding association matrix, and the appearance affinity is calculated by cosine similarity. m is the motion association matrix, and the motion affinity is calculated by Mahalanobis distance.
[0130] After matching, the real-time detection results of the obstacle with ID id in the video frame around the kth moment are:
[0131] [x id,l,k ,y id,l,k , w id,l,k ,h id,l,k , e id,l,k , x id,r,k ,y id,r,k , w id,r,k ,h id,r,k , e id,r,k ]
[0132] Through the above scheme, this embodiment matches the ID of the same obstacle in the video data obtained by the left and right cameras respectively through the Hungarian matching algorithm, and changes the ID of the obstacle in the video data obtained by the right camera to the ID of the obstacle in the video data obtained by the left camera; the ID of the video data can be unified, which facilitates subsequent data processing and improves the speed and efficiency of autonomous driving water surface scene perception.
[0133] Further, Figure 6 FIG. 5 is a flow chart of a fifth embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention. Figure 6 As shown, a fifth embodiment of the method for sensing a water surface scene in an autonomous driving system of the present invention is proposed based on the first embodiment. In this embodiment, step S30 specifically includes the following steps:
[0134] Step S31, stereo matching is performed on pixel data within detection frames of the obstacle in the left and right video frames at the same time in the segmentation result by using a semi-global matching (SGM) algorithm to obtain a disparity map based on the left camera.
[0135] It should be noted that, by performing stereo matching on the pixel data within the detection frame of the obstacle in the left and right video frames at the same time through the SGM algorithm, a disparity map based on the left camera can be obtained.
[0136] In the specific implementation, the process of calculating the disparity map by the SGM algorithm is as follows: calculate the matching cost between the to-be-matched pixel points in the left image and the candidate pixel points in the right image and perform cost aggregation, then select the disparity with the minimum cost value among all the cost values of the to-be-matched pixel points under all disparities as the optimal disparity, and then fit a univariate quadratic curve through the cost values under the optimal disparity and its two previous and subsequent disparities, take the disparity value represented by its minimum point as the optimal disparity of the pixel, and obtain the disparity map of the left image.
[0137] For pixel p, if the disparity is d, the matching cost is:
[0138]
[0139]
[0140]
[0141]
[0142] Among them, I b is the reference image, f D (I m ) is the matching image after image correction, I bp is the gray value of pixel p in the reference image, I mqTo match the gray value of pixel q in the image with a disparity of d with pixel p, is the mutual information between pixel p and pixel q. For image I 1 , I 2 The joint distribution diagram of is the value of the i-th row and k-th column in the joint distribution graph, for The sum of the values in the i-th row of , for The sum of the values in the kth column, g(i, k) is a Gaussian smoothing filter, and n is the number of matching left and right pixel pairs.
[0143] For pixel p, if the disparity is taken as d, the cost after cost aggregation is:
[0144]
[0145]
[0146] Among them, r is the direction, generally 8 directions, C(p, d) is the current matching cost, L r (pr, d) is the cost when the disparity of the previous pixel in the r direction remains unchanged, L r (pr, d-1) is the cost when the disparity of the previous pixel in the r direction is less than 1, L r (pr, d+1) is the cost when the disparity of the previous pixel in the r direction is greater than 1, L r (pr,i),L r (pr, k) is the cost of the previous pixel disparity in the r direction being greater than 1 compared to the current disparity.
[0147] The disparity map based on the left camera obtained by the SGM algorithm is:
[0148] {d u,v,k}
[0149] d u,v,k =x u,v,l,k -x u,v,r,k
[0150] Among them, d u,v,k is the disparity of the pixel in the uth row and vth column in the left video frame at the kth moment, that is, the difference in the x-axis direction of the matching pixel points on the imaging plane of the left and right cameras, x u,v,l,k , x u,v,r,k are the horizontal coordinates of the pixel on the imaging planes of the left and right cameras respectively.
[0151] Step S32: remove background pixels in the disparity map, and calculate the three-dimensional coordinates of non-background pixels in the camera coordinate system of the left camera by epipolar geometry.
[0152] It should be understood that after removing the background pixels, the three-dimensional coordinates of the non-background pixels in the camera coordinate system of the left camera can be calculated by epipolar geometry.
[0153] In the specific implementation, the three-dimensional coordinates of the non-background pixel points calculated by epipolar geometry in the camera coordinate system of the left camera are:
[0154]
[0155] Among them, x u,v,k,lc ,y u,v,k,lc , z u,v,k,lc is the 3D coordinate of the pixel in the uth row and vth column of the left video frame at the kth moment in the left camera coordinate system, x u,v,l,k ,y u,v,l,k are the horizontal and vertical coordinates of the pixel in the uth row and vth column of the left video frame at the kth moment on the imaging plane of the left camera, f is the focal length of the camera, and b is the baseline of the left and right cameras of the binocular camera.
[0156] Step S33: performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation based on the point set, and driving the gimbal to focus on key positions.
[0157] It can be understood that after the three-dimensional coordinates are transformed, the coordinates in the scene coordinate system can be obtained, and each obstacle can be represented by a point set, and then the global scene representation can be obtained based on the point set to drive the gimbal to focus on the key position.
[0158] Through the above scheme, this embodiment uses a semi-global matching SGM algorithm to perform stereo matching on the pixel data in the detection box of the obstacle in the left and right video frames at the same time in the segmentation result, and obtains a disparity map based on the left camera; the background pixels in the disparity map are eliminated, and the three-dimensional coordinates of the non-background pixels in the camera coordinate system of the left camera are calculated by epipolar geometry; the three-dimensional coordinates are converted to obtain a point set in the scene coordinate system, and a global scene representation is obtained according to the point set to drive the gimbal to focus on the key position; it can realize panoramic perception of a single binocular camera, and a closed-loop system with feedback ensures automatic perception of the perception platform, greatly saves manpower and material costs, maintains the unification of the scene data coordinate system in the water surface scene, is conducive to the expansion of autonomous driving technology to the water surface field, improves the accuracy of scene perception, reduces the probability of autonomous driving in water surface scenes colliding with obstacles, avoids accident risks, ensures the safety of autonomous driving, and improves the speed and efficiency of autonomous driving surface scene perception.
[0159] Further, Figure 7 FIG. 6 is a flow chart of a sixth embodiment of the method for sensing water surface scenes in an autonomous driving system according to the present invention. Figure 7 As shown, a sixth embodiment of the method for sensing water surface scenes of autonomous driving according to the present invention is proposed based on the fifth embodiment. In this embodiment, step S33 specifically includes the following steps:
[0160] Step S331, collecting the inertial measurement output results of the vehicle gimbal at multiple times.
[0161] It should be noted that the inertial measurement IMU output at multiple times may be collected, and the inertial measurement output results are generally the rotation quaternion of the gimbal and the longitude, latitude and altitude of the IMU at multiple times.
[0162] Step S332: performing coordinate transformation on the three-dimensional coordinates according to the inertial measurement output result to obtain a point set corresponding to the coordinates of the obstacle pixel points in the scene coordinate system.
[0163] It can be understood that the three-dimensional coordinates can be transformed through the inertial measurement output result to obtain the coordinates of the obstacle pixel points in the scene coordinate system, and each obstacle can be represented by a point set.
[0164] In the specific implementation, the transformation from the camera coordinate system to the gimbal coordinate system is:
[0165]
[0166] in, is the homogeneous coordinates of a point in the left camera coordinate system [x lc ,y lc , z lc , 1] T , is the homogeneous coordinate of the point in the gimbal coordinate system [x p ,y p , z p , 1] T . T lc,p is the transformation matrix from the left camera coordinate system to the gimbal coordinate system, which consists of one rotation and one translation, and is expanded as follows:
[0167]
[0168]
[0169] Among them, R lc,p is the rotation matrix, t lc,p is the translation matrix, b is the baseline length between the left and right cameras of the binocular camera, and l is the distance between the baseline and the origin of the gimbal coordinate system.
[0170] The transformation from the gimbal coordinate system to the ENU coordinate system with the IMU as the origin is:
[0171]
[0172] in, is the secondary coordinate of a point in the gimbal coordinate system [x p ,y p , z p ,1]T, is the homogeneous coordinates of the point in the ENU coordinate system with the IMU as the origin [x ie ,y ie , z ie , 1] T . T p,ie It is the transformation matrix from the gimbal coordinate system to the ENU coordinate system with the IMU as the origin, which consists of one rotation and is expanded as follows:
[0173]
[0174]
[0175] Among them, R p,ie is the rotation matrix, q=[q 0 ,q 1 ,q 2 ,q 3 ] is the quaternion output by the IMU that represents the real-time rotation of the gimbal in three-dimensional space.
[0176] Correspondingly, the two coordinate systems are expressed as:
[0177] S meter =[k, origin]
[0178] S pixel =[k, origin, w, h, res]
[0179] Among them, S meter is the scene coordinate system, S pixel is the scene pixel coordinate system, k is the time when the two coordinate systems are initialized, origin is the latitude, longitude and elevation of the IMU at time k, that is, the origin of the scene coordinate system, w and h are the width and height of the scene pixel coordinate system, and res is the resolution of the scene pixel coordinate system, that is, each pixel in this coordinate system represents res meters.
[0180] The transformation from scene coordinate system to scene pixel coordinate system is:
[0181]
[0182] x∈[-w / 2, w / 2], y∈[-h / 2, h / 2]
[0183] Where [x, y, 1] Tis the homogeneous coordinates of a point in the scene coordinate system, [u, v, 1] T is the homogeneous coordinate of the point in the scene pixel coordinate system, u represents the uth column, and v represents the vth row.
[0184] Accordingly, the output of the IMU at the kth moment is:
[0185] [q 0,k ,q 1,k ,q 2,k ,q 3,k ,lon k ,lat k , alt k ]
[0186] Among them, q 0,k ,q 1,k ,q 2,k ,q 3,k is the quaternion representing the rotation of the platform coordinate system at the kth moment relative to the ENU coordinate system with the IMU as the origin, lon k ,lat k , alt k is the longitude, latitude and altitude of the IMU at the kth moment.
[0187] The coordinates of the obstacle pixel point are transformed from the left camera coordinate system to the ENU coordinate system with the IMU as the origin:
[0188]
[0189]
[0190]
[0191] in, is the homogeneous coordinate of the pixel in the uth row and vth column of the left video frame at the kth moment in the left camera coordinate system, is the homogeneous coordinate of the pixel in the uth row and vth column of the left video frame at the kth time in the ENU coordinate system with the IMU as the origin, T lc,p is the transformation matrix from the left camera coordinate system to the gimbal coordinate system, T p,ie,k It is the transformation matrix from the gimbal coordinate system obtained according to the quaternion output by the IMU at the kth moment to the ENU coordinate system with the IMU as the origin.
[0192] The coordinates of the obstacle pixel point are transformed from the ENU coordinate system with the IMU as the origin to the scene coordinate system:
[0193]
[0194]
[0195]
[0196]
[0197]
[0198] Among them, x u,v,k,sm ,y u,v,k,sm , z u,v,k,sm is the coordinate of the pixel in the uth row and vth column of the left video frame at the kth moment in the scene coordinate system, x u,v,k,ecef ,y u,v,k,ecef , z u,v,k,ecef is the coordinate of the pixel in the ECEF coordinate system, x u,v,k,ie ,y u,v,k,ie , z u,v,k,ie is the coordinate of the pixel in the ENU coordinate system with IMU as the origin, x ieo,k,ecef ,y ieo,k,ecef , z ieo,k,ecef is the coordinate of IMU in ECEF coordinate system at the kth moment, lon ieo,k,lla ,lat ieo,k,lla , alt ieo,k,lla is the latitude, longitude and altitude of the IMU at the kth moment, x so,ecef ,y so,ecef , z so,ecef is the coordinate of the origin of the scene coordinate system in the ECEF coordinate system, lon so,lla ,lat so,lla , alt so,lla is the latitude, longitude and elevation of the origin of the scene coordinate system, e 2 is the eccentricity of the ellipsoid, and N is the radius of curvature of the reference ellipsoid.
[0199] Then all the pixel points of the obstacle with ID id in the video frame at the kth moment are expressed in the scene coordinate system as follows:
[0200] {a i,id,k,sm}
[0201] a i,id,k,sm =[x i,id,k,sm ,y i,id,k,sm , z i,id,k,sm ] T
[0202] Among them, a i,id,k,sm is the coordinate of the i-th point of the obstacle with ID id at the k-th moment in the scene coordinate system.
[0203] Step S333: obtaining a global scene representation according to the point set, and driving the gimbal to focus on key positions.
[0204] It should be understood that the global scene representation can be obtained through the point set, thereby driving the gimbal to focus on key locations.
[0205] Furthermore, the step S333 specifically includes the following steps:
[0206] Obtaining the obstacle ID corresponding to the point set, and calculating the convex hull according to the obstacle ID;
[0207] Calculate the current speed of each obstacle at the kth moment according to the position change of the centroid of the convex hull, and estimate the predicted centroid and predicted speed of each obstacle at the next moment according to the current speed and Kalman filter;
[0208] When the current system mode is the attention mode, the coordinates of the convex hull are converted from the scene coordinate system to the scene pixel coordinate system to obtain a global scene representation, and the target obstacle with the highest collision risk value is calculated according to the global scene representation;
[0209] The gimbal is driven to turn according to the predicted center of mass, the predicted speed and the target obstacle and focuses on the key position with the highest risk value.
[0210] It should be noted that the point set of the obstacle with ID id in the point set at the kth moment in the scene coordinate system can be used to calculate the convex hull, and the speed of the obstacle with ID id at the kth moment can be calculated based on the position transformation of the centroid of the convex hull. The center of mass and speed of the obstacle with ID id at the kth moment can be estimated based on the Kalman filter, and the center of mass and speed at the next moment can be predicted.
[0211] It can be understood that the coordinates of the convex hull point of the obstacle with ID id at the kth moment are converted from the scene coordinate system to the scene pixel coordinate system, the representation of the obstacle with ID id detected at the kth moment is obtained, and the global scene representation is updated.
[0212] It should be understood that in actual operation, Figure 8 As shown, Figure 8 This is a schematic diagram of the closed-loop system of the automatic driving surface scene perception method of the present invention. The system mode at the current moment k can be judged. If it is in the panoramic mode, the panoramic scan continues; if it is in the attention mode, the obstacle with the highest risk value is calculated according to the global scene representation, and its position is used as the position that needs to be focused on at the next moment. The gimbal rotation angle is calculated and sent to the gimbal to drive the gimbal to turn to the obstacle to complete the closed loop.
[0213] In the specific implementation, the Graham scanning method can be used to calculate the convex hull of the point set. The specific process is: after sorting with the horizontal coordinate as the first keyword and the vertical coordinate as the second keyword, the point in the lower left corner is obtained, and this point must be on the convex hull; taking this point as the pole, the other points in the point set are sorted by polar angles, and the points sorted by polar angles are connected in sequence to obtain a polygon that surrounds all the points; then maintain a stack and traverse each point in the order of polar angle sorting: if the number of points in the stack is less than 3, push them directly into the stack, otherwise, check whether the rotation direction of the two vectors composed of the three points at the top of the stack is counterclockwise, if so, push them into the stack, if not, pop the top of the stack, until the number of points in the stack is less than 3 or the counterclockwise condition is met.
[0214] The convex hull of the obstacle with ID id at the kth moment is:
[0215] {a′ i,id,k,sm}
[0216] Among them, a′ i,id,k,sm is the coordinate of the i-th point on the convex hull of the obstacle with ID id at the k-th moment in the scene coordinate system.
[0217] The centroid of the convex hull of the obstacle with ID id at the kth moment is:
[0218]
[0219] Among them, cx id.k,sm ,cy id.k,sm are the horizontal and vertical coordinates of the convex hull centroid of the obstacle with ID id at the kth moment in the scene coordinate system, x′ i,id,k,sm , y′ i,id,k,sm are the horizontal and vertical coordinates of the i-th point on the convex hull in the scene coordinate system, and n is the number of points on the convex hull.
[0220] The speed of the obstacle with ID id at the kth moment is:
[0221]
[0222] Among them, vx id,k ,vy id,k are the speed of the obstacle with ID id on the x-axis and y-axis at the kth moment, lk is the last recorded time at the kth moment, cx id,k,sm ,cy id,k,sm are the horizontal and vertical coordinates of the convex hull particle of the obstacle with ID id at the kth moment in the scene coordinate system, They are respectively the horizontal and vertical coordinates of the center of mass of the obstacle with ID id in the scene coordinate system at the lkth moment.
[0223] Accordingly, the center of mass and velocity of the obstacle with ID id at the kth moment are estimated by Kalman filtering as follows:
[0224]
[0225] P′ id,k =P id,k -KHP id,k
[0226] K=P id,k H T (HP id,k H T +R id,k ) -1
[0227]
[0228]
[0229]
[0230]
[0231] H=1
[0232] in, is the best estimate of the center of mass and velocity of the obstacle with ID id at the kth moment, is the best estimate of the center of mass and velocity of the obstacle with ID id at the lkth moment, is the predicted value of the center of mass and speed of the obstacle with ID id at the kth moment, is the measured value of the convex hull particle and velocity of the obstacle with ID id at the kth moment, F is the change matrix of uniform linear motion, H is the mapping matrix, P′ id,k is the estimated covariance matrix of the obstacle with ID id at the kth moment, P id,k is the predicted covariance matrix of the obstacle with ID id at the kth moment, R id,k is the measurement covariance matrix of the obstacle with ID id at the kth moment, and K is the Kalman gain.
[0233] According to the Kalman filter, the center of mass and speed of the obstacle with ID id at the nkth moment are predicted as follows:
[0234]
[0235] Among them, nk is the next recording time of k.
[0236] The convex hull of the obstacle with ID id at the kth moment is expressed in the scene pixel coordinate system as follows:
[0237] {u i,id,k,sp , v i,id,k,sp}
[0238]
[0239] Among them, u i,id,k,sp , v i,id,k,sp is the convex hull point i of the obstacle with ID id at the kth moment in the scene pixel coordinate system at the uth row and vth column.
[0240] The obstacle with ID id at the kth moment is represented as:
[0241]
[0242] Among them, type id The type of obstacle with ID id. is the state estimate of the obstacle with ID id at the kth moment, namely, the center of mass and speed, {u i,id,k,sp , v i,id,k,sp} is the convex hull of the obstacle with ID id at the kth moment in the scene pixel coordinate system.
[0243] Accordingly, the global scene at the kth moment is expressed as:
[0244] scene k =[origin k ,w,h,res,{obstacle id,k}] T
[0245] Among them, origin k is the latitude, longitude and elevation of the origin of the scene coordinate system at the kth moment, w and h are the width and height of the scene pixel coordinate system, res is the resolution of the scene pixel coordinate system, {obstacle id,k} is the obstacle set at the kth moment.
[0246] Accordingly, the system mode is divided into a panoramic mode and an attention mode: in the panoramic mode, the binocular camera obtains scene information with a field of view of 90° each time, and the gimbal rotates three times to complete a circle; in the attention mode, the gimbal is driven to rotate so that the binocular camera only focuses on the direction with the highest risk value.
[0247] The time interval for switching to panoramic mode for panoramic scanning is:
[0248]
[0249] Among them, x ieo,k,sm ,y ieo,k,sm is the horizontal and vertical coordinates of the IMU at the kth moment in the scene coordinate system, vx k ,vy kis the speed of the system itself on the x-axis and y-axis at the kth moment, v′ is a preset speed parameter, and T represents the time required for a collision to occur if a new obstacle appears.
[0250] Correspondingly, the risk value function of the obstacle with ID id at the kth moment is:
[0251]
[0252]
[0253]
[0254]
[0255]
[0256]
[0257] Among them, f 1 , f 2 , f 3 is the risk factor, cost angle,id,k is the direction risk, cost dist,id,k is the distance risk, cost vel,id,k For speed risk, is the velocity vector of the system itself at the kth moment, is the position vector of the obstacle with ID id at the kth moment relative to the IMU in the scene coordinate system, is the projection of the velocity vector of the obstacle with ID id at the kth moment on the velocity vector of the system itself.
[0258] Correspondingly, the rotation angle of the gimbal at the kth moment in the attention mode is:
[0259]
[0260]
[0261] T ie,p =T p,ie T
[0262]
[0263]
[0264]
[0265] Among them, θ′ is the clockwise rotation angle of the gimbal at the kth moment, is the homogeneous coordinate of the center of mass of the obstacle with ID id in the gimbal coordinate system, is the homogeneous coordinate of the center of mass of the obstacle with ID id in the ENU coordinate system with IMU as the origin, T ie,p The transformation matrix from the ENU coordinate system with the IMU as the origin to the gimbal coordinate system.
[0266] Through the above scheme, this embodiment collects the inertial measurement output results of the vehicle gimbal at multiple times; performs coordinate transformation on the three-dimensional coordinates according to the inertial measurement output results to obtain a point set corresponding to the coordinates of the obstacle pixel points in the scene coordinate system; obtains a global scene representation according to the point set to drive the gimbal to focus on key directions; reduces the probability of autonomous driving in water surface scenes colliding with obstacles, avoids accident risks, ensures the safety of autonomous driving, and improves the speed and efficiency of autonomous driving surface scene perception.
[0267] Accordingly, the present invention further provides an autonomous driving water surface scene perception device.
[0268] Reference Fig. 9 , Fig. 9 This is a functional module diagram of the first embodiment of the autonomous driving water surface scene perception device of the present invention.
[0269] In a first embodiment of the automatic driving water surface scene perception device of the present invention, the automatic driving water surface scene perception device comprises:
[0270] The data acquisition module 10 is used to acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire the trained multi-target tracking network.
[0271] The segmentation module 20 is used to obtain real-time video data of the water surface scene using a vehicle gimbal equipped with a binocular camera, and perform obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result.
[0272] The scene representation and understanding module 30 is used to obtain the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, perform coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtain a global scene representation based on the point set, and drive the gimbal to focus on the key position.
[0273] Among them, the steps implemented by each functional module of the autonomous driving water surface scene perception device can refer to the various embodiments of the autonomous driving water surface scene perception method of the present invention, and will not be repeated here.
[0274] In addition, an embodiment of the present invention further proposes a storage medium, on which an autonomous driving surface scene perception program is stored. When the autonomous driving surface scene perception program is executed by a processor, the operations in the embodiment of the autonomous driving surface scene perception method described above are implemented.
[0275] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0276] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0277] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for sensing water surface scenes for autonomous driving, characterized in that: The method for water surface perception in an autonomous driving scene includes: Acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire a trained multi-target tracking network; Using a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, and performing obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result; Obtain the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, perform coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtain a global scene representation according to the point set, and drive the gimbal to focus on the key position; The method of obtaining the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation according to the point set, and driving the gimbal to focus on the key position includes: Performing stereo matching on pixel data within detection frames of the obstacle in the left and right video frames at the same time in the segmentation result by using a semi-global matching (SGM) algorithm to obtain a disparity map based on the left camera; Eliminate the background pixels in the disparity map, and calculate the three-dimensional coordinates of the non-background pixels in the camera coordinate system of the left camera by epipolar geometry; Performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation based on the point set, and driving the gimbal to focus on key positions; The step of performing coordinate transformation on the three-dimensional coordinates to obtain a point set in a scene coordinate system, obtaining a global scene representation according to the point set, and driving the gimbal to focus on a key position includes: Collect the inertial measurement output results of the vehicle gimbal at multiple times; Performing coordinate transformation on the three-dimensional coordinates according to the inertial measurement output result to obtain a point set corresponding to the coordinates of the obstacle pixel points in the scene coordinate system; Obtaining a global scene representation based on the point set, and driving the gimbal to focus on key locations; The step of obtaining a global scene representation according to the point set and driving the gimbal to focus on a key position includes: Obtaining the obstacle ID corresponding to the point set, and calculating the convex hull according to the obstacle ID; Calculate the current speed of each obstacle at the kth moment according to the position change of the centroid of the convex hull, and estimate the predicted centroid and predicted speed of each obstacle at the next moment according to the current speed and Kalman filter; When the current system mode is the attention mode, the coordinates of the convex hull are converted from the scene coordinate system to the scene pixel coordinate system to obtain a global scene representation, and the target obstacle with the highest collision risk value is calculated according to the global scene representation; The gimbal is driven to turn according to the predicted center of mass, the predicted speed and the target obstacle and focuses on the key position with the highest risk value.
2. The method for sensing water surface scenes for autonomous driving according to claim 1, characterized in that: The step of acquiring historical video data of a water surface scene, training an initial multi-target tracking network according to the historical video data, and acquiring a trained multi-target tracking network comprises: Acquire channel videos of water surface scenes at different time periods, manually annotate and mask the channel videos, and obtain historical video data in which channel obstacles are marked with boxes and masked with instances in the channel videos; An initial multi-object tracking network with instance segmentation branches added is trained according to the historical video data to obtain a trained multi-object tracking network.
3. The method for sensing water surface scenes for autonomous driving according to claim 1, characterized in that: The method uses a vehicle gimbal equipped with a binocular camera to obtain real-time video data of the water surface scene, and performs obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result, including: The binocular camera and the inertial measurement unit (IMU) are fixedly mounted on the vehicle gimbal, and the scene coordinate system and the scene pixel coordinate system are initialized; Acquiring real-time video data of the water surface scene using the binocular camera; Obstacle tracking and instance segmentation are performed on the real-time video data according to the trained multi-target tracking network to obtain segmentation results of real-time obstacle detection at multiple moments in the video data obtained by the left and right cameras respectively.
4. The method for sensing water surface scenes for autonomous driving according to claim 3, characterized in that: After performing obstacle tracking and instance segmentation on the real-time video data according to the trained multi-target tracking network to obtain segmentation results of obstacle real-time detection at multiple moments in the video data obtained by the left and right cameras respectively, the autonomous driving water surface scene perception method further includes: The IDs of the same obstacle in the video data obtained by the left and right cameras are matched by the Hungarian matching algorithm, and the ID of the obstacle in the video data obtained by the right camera is changed to the ID of the obstacle in the video data obtained by the left camera.
5. An autonomous driving water scene perception device, characterized in that: The autonomous driving water surface scene perception device comprises: A data acquisition module, used to acquire historical video data of a water surface scene, train an initial multi-target tracking network according to the historical video data, and acquire a trained multi-target tracking network; A segmentation module is used to obtain real-time video data of the water surface scene using a vehicle gimbal equipped with a binocular camera, and perform obstacle tracking and instance segmentation on the real-time video data according to a trained multi-target tracking network to obtain a segmentation result; A scene representation and understanding module is used to obtain the three-dimensional coordinates corresponding to the non-background pixel points in the segmentation result according to the semi-global matching SGM algorithm, perform coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtain a global scene representation according to the point set, and drive the gimbal to focus on the key position; The scene representation and understanding module is further used to perform stereo matching on the pixel data in the detection frame of the obstacle in the left and right video frames at the same time in the segmentation result through a semi-global matching SGM algorithm to obtain a disparity map based on the left camera; remove the background pixel points in the disparity map, and calculate the three-dimensional coordinates of the non-background pixel points in the camera coordinate system of the left camera through epipolar geometry; perform coordinate transformation on the three-dimensional coordinates to obtain a point set in the scene coordinate system, obtain a global scene representation based on the point set, and drive the gimbal to focus on the key position; The scene representation and understanding module is also used to collect the inertial measurement output results of the vehicle gimbal at multiple times; perform coordinate transformation on the three-dimensional coordinates according to the inertial measurement output results to obtain a point set corresponding to the coordinates of the obstacle pixel points in the scene coordinate system; obtain a global scene representation according to the point set to drive the gimbal to focus on the key position; The scene representation and understanding module is also used to obtain the obstacle ID corresponding to the point set, and calculate the convex hull according to the obstacle ID; calculate the current speed of each obstacle at the kth moment according to the position change of the center of mass of the convex hull, and estimate the predicted center of mass and predicted speed of each obstacle at the next moment according to the current speed and Kalman filtering; when the current system mode is the attention mode, convert the coordinates of the convex hull from the scene coordinate system to the scene pixel coordinate system to obtain a global scene representation, and calculate the target obstacle with the highest collision risk value according to the global scene representation; drive the gimbal to turn according to the predicted center of mass, the predicted speed and the target obstacle, and pay attention to the key position with the highest risk value.
6. An autonomous driving water scene perception device, characterized in that: The autonomous driving water surface scene perception device includes: a memory, a processor, and an autonomous driving water surface scene perception program stored in the memory and executable on the processor, wherein the autonomous driving water surface scene perception program is configured to implement the steps of the autonomous driving water surface scene perception method as described in any one of claims 1 to 4.
7. A storage medium, characterized in that: The storage medium stores an autonomous driving surface scene perception program, which, when executed by a processor, implements the steps of the autonomous driving surface scene perception method according to any one of claims 1 to 4.
