Robust Pure Vision Dynamic Target Segmentation Method, System and Storage Medium for Intelligent Vehicle

Abstract

The present invention provides a robust pure vision dynamic target segmentation method for an intelligent vehicle, which includes the following steps: S1. Perform ROF calibration on the field of view space; specifically: S11. Set the camera hardware parameters and the minimum target movement speed; S12. Perform calibration on the spatial residual flow of the moving target; S13. Obtain the minimum residual flow in different regions of the field of view; S2. Detect candidate moving targets and calculate the camera self-movement parameters; S3. Judge the motion state; specifically: S31. Calculate the residual flow of the candidate moving target; S32. Perform randomness judgment on the residual flow direction and threshold constraint judgment on the residual flow to determine the motion state of the candidate moving target; S33. Output the motion state. The present invention can improve the detection accuracy and recall rate of pure vision moving target segmentation of the intelligent vehicle, and further improve the dynamic environment perception ability of the intelligent vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of visual dynamic target segmentation, in particular to a robust intelligent vehicle pure vision dynamic target segmentation method, system and storage medium. Background Art

[0002] The traffic environment faced by intelligent vehicles during road driving is extremely complex, not only including various static obstacles, but also having highly dynamic and random traffic participants on the road. When an intelligent vehicle is driving on the road, it not only needs to pay attention to whether there are obstacles and other traffic participants around, but also needs to perceive the motion state of traffic participants in order to make the best decisions and avoid collisions. Among the numerous sensors installed on intelligent vehicles, binocular cameras have the characteristics of low cost and rich scene information acquisition, and have received extensive attention from many scholars in the field of computer vision and intelligent vehicle manufacturers. A robust intelligent vehicle pure vision dynamic target segmentation method is of great significance for improving the performance of intelligent vehicle environment perception.

[0003] Existing intelligent vehicle vision moving target segmentation methods can be mainly divided into two categories. One is based on traditional vision methods, and the other is based on deep learning.

[0004] Traditional vision methods: Motion target detection methods based on geometric constraints usually extract feature points from images, and then construct geometric constraint relationships satisfied by static regions through feature points in two or multiple images. Points that satisfy the constraints are regarded as static points, and vice versa as moving points. These geometric constraint relationships include affine transformation, epipolar constraint and homography transformation, as well as multi-view constraint relationships between four images in two consecutive frames, etc.

[0005] Deep learning methods: Different researchers have different deep learning models for motion object segmentation. The inputs of the network usually include: binocular images, background images; the outputs include depth maps, optical flow maps, scene flow maps, and scene instances. In 2017, Heo et al. proposed a method for detecting moving objects under dynamic background conditions using a convolutional neural network. The proposed method consists of two deep learning networks, namely the Appearance net (A-Net) and the Motion net (M-Net). The purpose of A-Net is to detect the appearance of moving objects, and M-Net detects their motion states. Finally, the two networks are combined to detect moving objects. Siam et al. combined the tasks of vehicle detection and motion segmentation and proposed a method for traffic scene understanding by combining appearance and motion features. The network structure is a two-step process. First, a two-channel VGG16 network is used to learn and merge the features of RGB images and dense optical flow images, and then vehicle detection and moving vehicle segmentation are performed in sequence. Muthu et al. proposed a method for motion object segmentation based on an RGB-D camera, which considers multiple moving objects, combines semantic appearance and motion information, and focuses on small and slowly moving targets. Li et al. used a convolutional neural network to estimate the 3D pose of a car and optimized the camera pose while estimating the car pose. This method improves the accuracy of 6-degree-of-freedom object pose estimation but loses generality. Rashed et al. proposed a deep learning model for motion object detection. Inputting the motion information of the vehicle into the deep learning model enables the model to implicitly compensate for the self-motion of the camera to improve the performance of the system. Summary of the Invention

[0006] The present invention provides a robust intelligent vehicle pure vision dynamic target segmentation method, system, and storage medium. The method first uses a deep learning instance segmentation method (such as SOLOv2) to segment potential moving targets in the scene; then sequentially determines the motion states of each potential moving target according to certain criteria. The proposed criteria are based on the differences in the residual optical flow direction and amplitude between moving targets and stationary targets. The residual optical flow direction of stationary targets has more obvious randomness, and the residual optical flow amplitude is often larger. Based on this difference, a residual optical flow direction randomness judgment criterion and a residual optical flow threshold constraint criterion are designed. When a candidate moving target meets the above two criteria, it is regarded as a moving target.

[0007] To achieve the above object, the first aspect of the embodiments of this specification discloses a robust intelligent vehicle pure vision dynamic target segmentation method, including the following steps:

[0008] S1. Perform FOV space ROF calibration; specifically:

[0009] S11. Set the camera hardware parameters and the target minimum motion speed;

[0010] S12. Calibrate the spatial residual flow of the moving target;

[0011] S13. Obtain the minimum residual flow in different regions of the field of view;

[0012] S2. Perform candidate moving target detection and camera self - motion parameter calculation;

[0013] S3. Judge the motion state of the candidate moving target; specifically:

[0014] S31. Calculate the residual flow of the feature points of the candidate moving target;

[0015] S32. Judge the randomness of the residual flow direction and the threshold constraint of the residual flow to determine the motion state of the candidate moving target;

[0016] S33. Output the motion state.

[0017] In some embodiments, S11 includes:

[0018] S111. Set the camera hardware parameters: focal length f, principal point coordinates (cx, cy), binocular camera baseline d, camera resolution H * W, where H represents the height of the image and W represents the width of the image;

[0019] S112. The world coordinate system is established on the left camera, with the camera optical center as the origin of the world coordinate system. The positive direction of X is horizontally to the right of the image plane, the positive direction of Y is horizontally downward of the image plane, and the positive direction along the optical axis is Z. In this coordinate system, the minimum motion amount parameter of the target between two adjacent frames is: motion = [ΔX, ΔY, ΔZ]; ΔX, ΔY, and ΔZ respectively represent the motion amounts of the target in the X, Y, and Z directions, and the unit can be m;

[0020] S113. The range of the target in the field of view refers to the forward range and distance that the intelligent vehicle needs to detect according to the actual engineering situation. Design a cuboid area as the area where the target may be, and the cuboid area is x ∈ [xmin, xmax]; y ∈ [ymin, ymax]; z ∈ [zmin, zmax]; for example, x ∈ [-20, 20]; y ∈ [-20, 5]; z ∈ [2, 50] can be taken, with the unit of m. Sample the cuboid area at a certain step size ε to obtain a set of sampling points, and use this set of sampling points to simulate the surface feature points of the moving target in the field of view. The three - dimensional coordinates of the surface feature points are denoted as X t-1 , where the subscript t - 1 represents the time of t - 1.

[0021] In some embodiments, S12 includes:

[0022] S121. Given the target motion amount motion = [Δx, ΔY, ΔZ];

[0023] S122. Given the camera self - motion parameters tr = [R, T], where R is the rotation matrix of the camera at adjacent moments and T is the translation vector of the camera at adjacent moments; calculate X′ t = RX t-1 + t, and project X′ t onto the left image at time t and denote it as img_ideal_static_t;

[0024] S123. Calculate the displacement after the target motion: X mt = X′ t + motion;

[0025] S124. Project X mt onto the left camera at time t to obtain the image point coordinates, denoted as img_ideal_dynamic_t;

[0026] The residual optical flow ROF = img_ideal_static_t - img_ideal_dynamic_t;

[0027] S125. Calculation of the ROF values of object points in different regions within the field of view: According to the possible range of the target to be detected in the field of view in actual engineering, divide the object point distances into multiple regions. For example, divide them into 6 intervals: 0 - 5m, 5 - 10m, 10 - 15m, 15 - 20m, 20 - 25m, 25 - 50m. At the same time, divide the image plane into multiple regions; statistically analyze the ROF set calculated in step S124 according to the distance and different regions in the image plane to obtain a ROF value statistical chart, and take the minimum value of the ROF values of each region at different distances as the final calibrated output value;

[0028] By changing the parameters of the camera hardware and the values of the three components in the target motion, motion, to simulate the corresponding relationship between the spatial target motion state and its residual optical flow value in the image plane under different cameras and different scenarios, the minimum value of the residual optical flow will be used as the threshold for judging the target motion state.

[0029] In some embodiments, in S31:

[0030] For the set of feature - matching points on four images calculate the object point coordinates P t-1 at time t - 1;

[0031] Calculate the predicted coordinates P′ of the object point P t-1 at time tt = RP t-1 + t;

[0032] Back-project the predicted coordinate P' t onto the image at time t to obtain the predicted image point. The difference between this predicted image point and the observed image point is the residual optical flow ROF.

[0033] In some embodiments, in S32:

[0034] Input: ROF of a single set of candidate moving target feature points, denoted as rofs, rofs = [rof1, rof2,..., rofi,... rofn], where rofi represents the residual optical flow of the i-th feature point, and n is the number of feature points;

[0035] Output: Whether it has randomness;

[0036] Specifically:

[0037] Calculate the residual optical flow angle of each feature point, denoted as ang, ang = [ang1, ang2,..., angi... angn], and the range of angi is [0 - 2π];

[0038] Sort each element in ang in ascending order;

[0039] Take the first 1 / 4 elements in ang and calculate their average value, denoted as ang_min; take the last 1 / 4 elements in ang and calculate their average value, denoted as ang_max;

[0040] Calculate the angle range ang_rang = ang_max - ang_min;

[0041] When ang_rang > π, ang_rang = ang_max + (2π - ang_min);

[0042] Determine whether it meets the direction randomness; if ang_rang > thresh_ang, it is considered that the set of feature points meets the direction consistency; thresh_ang is an empirical value.

[0043] In some embodiments, in S32:

[0044] For the ROF set of candidate moving target feature points, first calculate the average value ave_rof of the ROF set;

[0045] Calculate the average distance between the candidate moving target feature points and the camera, denoted as dis_ave;

[0046] Calculate the region of the image plane where the center of the candidate moving target feature points is located, denoted as area_index;

[0047] According to dis_ave and area_index, find the corresponding threshold in the ROF value statistical graph, denoted as thresh_rof;

[0048] If ave_rof > thresh_rof, then consider the candidate moving target as a moving target, otherwise as a stationary target.

[0049] The second aspect of the embodiments of this specification discloses a robust intelligent vehicle pure vision dynamic target segmentation system, including:

[0050] A field of view space ROF calibration module, used to set camera hardware parameters and the minimum moving speed of the target, perform calibration of the spatial residual flow of the moving target, obtain the minimum residual flow in different regions of the field of view, and use this as the threshold for judging the motion state;

[0051] A calculation module, used to detect candidate moving targets and calculate the camera self-motion parameters;

[0052] A motion state judgment module, used to calculate the residual flow of the candidate moving target, judge the randomness of the residual flow direction and the threshold constraint of the residual flow, and determine the motion state of the candidate moving target.

[0053] In some embodiments, the robust intelligent vehicle pure vision dynamic target segmentation system further includes:

[0054] A processor, respectively connected to the field of view space ROF calibration module, the calculation module and the motion state judgment module;

[0055] A memory, connected to the processor and storing a computer program that can run on the processor;

[0056] Wherein, when the processor executes the computer program, the processor controls the field of view space ROF calibration module, the calculation module and the motion state judgment module to work, so as to implement the above-mentioned robust intelligent vehicle pure vision dynamic target segmentation method.

[0057] The third aspect of the embodiments of this specification discloses a computer-readable storage medium, the storage medium stores computer instructions, and when the computer reads the computer instructions, the computer executes the above-mentioned robust intelligent vehicle pure vision dynamic target segmentation method.

[0058] In summary, the present invention has at least the following beneficial effects:

[0059] The present invention first uses a deep learning instance segmentation method (such as SOLOv2) to segment potential moving targets in a scene; then, according to certain criteria, it sequentially determines the motion states of each potential moving target. Compared with traditional geometric methods, the proposed method uses deep learning technology to first find possible moving targets - narrowing the search range of moving targets, greatly improving the search speed of moving targets and reducing the false detection rate of moving targets; compared with pure deep learning, multi-view geometric constraints are adopted, and the entire calculation process has a clear mathematical expression. When the moving target detection fails, the reason for the error can be traced step by step; the proposed moving target judgment criteria are based on the differences in the residual optical flow direction and amplitude between moving targets and stationary targets. The residual optical flow direction of stationary targets has more obvious randomness, and the residual optical flow amplitude is often larger. Based on this difference, a judgment criterion for the randomness of the residual optical flow direction and a threshold constraint criterion for the residual optical flow are designed. When designing the threshold, the highly dynamic and complex nature of traffic scenes is fully considered. Targets in the scene often have different distances, moving speeds, and moving directions. Using a simulation method, the residual optical flow of targets at different distances in different regions of the image plane is obtained under given hardware parameters and the minimum moving speed of the target, and it is used as the threshold, achieving high-accuracy and high-recall detection of moving targets in high-dynamic and high-complex traffic scenes, significantly improving the pure vision dynamic environment perception ability in the field of intelligent vehicles, and having important significance for improving the overall safety of pure vision intelligent vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0061] Figure 1 It is a flowchart of the robust intelligent vehicle pure vision dynamic target segmentation method involved in the present invention.

[0062] Figure 2 It is a schematic diagram of a stereo sequence image involved in the present invention.

[0063] Figure 3 It is a flowchart of the judgment of the target motion state involved in the present invention.

[0064] Figure 4 It is a schematic diagram of the modules of the robust intelligent vehicle pure vision dynamic target segmentation system involved in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] In the following text, only some exemplary embodiments are briefly described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and the description are considered to be exemplary in nature rather than restrictive.

[0066] The following disclosure provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0067] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0068] The general idea of the present invention is:

[0069] The process is as Figure 1 shown. The input of the process is the binocular sequence images at adjacent moments, and the output of the process is the moving targets in the scene. The core idea is to first detect the candidate moving targets in the scene using deep learning methods ( Figure 1 point 2 in Figure 1 ), then calculate the residual optical flow of the feature points of the candidate moving targets, and determine the motion state of the candidate moving targets according to the randomness of the residual optical flow direction and the comparison between the residual optical flow amplitude and the threshold ( Figure 1 point 3 in

[0070] ). In order to obtain a threshold with strong adaptability, it is necessary to calibrate the spatial residual flow of the moving targets in advance according to the adopted hardware and the possible minimum speed of the targets in the scene, and use the minimum residual flow in different regions of the field of view as the threshold (

[0071] point 1 in

[0072] ). In the above solution, point 2 can refer to the following or existing technologies. The main improvements of the present invention are points 1 and 3.

[0073] S1. Perform ROF calibration for the field of view space; specifically:

[0074] S11. Set the camera hardware parameters and the minimum moving speed of the target;

[0075] S12. Perform calibration of the spatial residual flow of the moving targets;

[0076] S2. Perform candidate moving target detection and camera self - motion parameter calculation;

[0077] S3. Perform motion state judgment; specifically:

[0078] S31. Calculate the residual flow of the candidate moving target;

[0079] S32. Perform randomness judgment of the residual flow direction and threshold constraint judgment of the residual flow to determine the motion state of the candidate moving target;

[0080] S33. Output the motion state.

[0081] In some embodiments, a binocular camera is generally mounted in front of an intelligent vehicle. During the driving process of the intelligent vehicle, scene images are collected in real - time, and stereo sequence images are as Figure 2 shown. Let respectively represent two sets of stereo images collected by the left and right cameras at times t - 1 and t. The imaging points of the spatial point P on are p t-1 and p t . At times t - 1 and t, the camera coordinate systems are O1X t-1 Y t-1 Z t-1 , O2X t Y t Z t respectively, and the transformation matrix (also known as the camera self - motion parameter) between the two camera coordinate systems is R, t.

[0082] t = [T x T y T z .

[0083] In some embodiments, for the field - of - view (FOV) space ROF calibration and the moving target space residual flow calibration, it is to obtain the residual optical flow of targets at different distances in space at different positions on the image plane under the given binocular hardware and target motion speed conditions. This residual optical flow value will be used for the judgment of the target motion state. The calibration is carried out in a simulation environment, and the specific calibration method is given below.

[0084] 1. Calibration preparation

[0085] 11. Camera hardware parameters

[0086] The camera hardware parameters include: focal length f, principal point coordinates (cx, cy), binocular camera baseline d, camera resolution H * W, where H represents the height of the image and W represents the width of the image.

[0087] 12. Minimum target motion speed

[0088] First, the world coordinate system is established on the left camera, with the camera optical center as the origin of the world coordinate system. The positive X-axis is horizontally to the right of the image plane, the positive Y-axis is horizontally downward of the image plane, and the positive Z-axis is along the optical axis direction. In this coordinate system, the minimum motion amount parameter of the target between two adjacent frames is: motion = [Δx, ΔY, ΔZ]; Δx, ΔY, and ΔZ respectively represent the motion amounts of the target in the X, Y, and Z directions, and the unit can be m.

[0089] 13. Range of the target in the field of view

[0090] The range of the target in the field of view refers to the front range and distance that the intelligent vehicle needs to detect according to the actual engineering situation. A cuboid area can be designed as the area where the target may be located. For example, the cuboid area is x ∈ [xmin, xmax]; y ∈ [ymin, ymax]; z ∈ [zmin, zmax]; for example, x ∈ [-20, 20]; y ∈ [-20, 5]; z ∈ [2, 50] can be taken, and the unit is m. Sample the cuboid area at a certain step size to obtain a set of sampling points. Use this set of sampling points to simulate the surface feature points of the moving target in the field of view. The three-dimensional coordinates of these surface feature points are denoted as X t-1 , and the subscript t - 1 represents the time at t - 1.

[0091] In some embodiments, the calibration steps of the motion target space residual flow:

[0092] Given the target motion amount motion = [Δx, ΔY, ΔZ];

[0093] Given the camera self-motion parameters tr = [R, T], calculate X' t = RX t-1 + T, and project X' t onto the left image at time t and denote it as img_ideal_static_t;

[0094] Calculate the displacement of the target after motion: X mt = X' t + motion;

[0095] Project X mt onto the left camera at time t to obtain the image point coordinates, denoted as img_ideal_dynamic_t, and the residual optical flow ROF = img_ideal_static_t - img_ideal_dynamic_t;

[0096] Calculation of the ROF values of object points in different regions within the field of view; according to the possible range of the target to be detected in the field of view in actual engineering, the object point distances are divided into multiple regions. For example, based on the inventor's experience, it can be divided into 6 intervals: 0 - 5m, 5 - 10m, 10 - 15m, 15 - 20m, 20 - 25m, and 25 - 50m. At the same time, the image plane is divided into multiple regions. For example, as shown in the following figure, the image plane is divided into 3×3 = 9 regions. The ROF set calculated in step S124 is statistically analyzed according to the distance and different regions in the image plane, and the minimum value of the ROF value in each region at different distances is used as the final output value for calibration. In this example, finally, 6 ROF value statistical charts will be obtained, corresponding to the 6 intervals of the object point distances respectively; each ROF value statistical chart is 3 rows and 3 columns with 9 numbers, representing the minimum value of the ROF corresponding to different regions of the image plane.

[0097] Example of the ROF value statistical chart: corresponding to the target distance of 0 - 5m, the three rows and three columns at the bottom of the figure represent the 9 regions of the image plane, and the number in each region represents the ROF threshold at the corresponding region of the image plane when the target distance is less than 5 meters.

[0098]

[0099] By changing the parameters of the camera hardware and the values of the three components in the target motion, the corresponding relationship between the motion state of the spatial target and its residual optical flow value in the image plane under different cameras and different scenarios can be simulated. The minimum value of the residual optical flow will be used as the threshold for judging the target motion state.

[0100] In some embodiments, detection of candidate moving targets and calculation of camera self - motion parameters:

[0101] 1. Detection of candidate moving targets

[0102] For Use the SOLOv2 algorithm or the MASKRCNN algorithm to segment the candidate moving targets (vehicles, pedestrians) in the scene.

[0103] 2. Calculation of camera self - motion parameters

[0104] The calculation principle of the camera self - motion parameters is as follows: When the feature matching points of a spatial point on 4 adjacent - frame images are known, denoted as The coordinates of these points in the O1X Y t-1 Y t-1 Z t-1 、O2X t Y t Z t coordinate systems can be calculated as the coordinates P t-1 (X t-1 ,Yt-1 , Z t-1 ), and P t (X t , Y t , Z t ). P t-1 and P t satisfies the camera self - motion parameter constraint of P t = RP t-1 + t. When there are 3 or more pairs of spatial points, R and t can be calculated by the least - squares method, the SVD method, the quaternion method, or the Newton descent method. For specific methods, please refer to the literature (Zhou D, Fremont V, Quost B, et al. On modeling ego - motion uncertainty for moving object detection from a mobile platform[C] / / Intelligent Vehicles Symposium, 2014:1332 - 1338.).

[0105] In some embodiments, motion state judgment:

[0106] The process of target motion state judgment is as Figure 3 shown. For any potential moving target, first calculate its feature points and the residual flow of the feature points; then judge whether the set of residual flows satisfies randomness. If it does, it is judged as a stationary target; if not, judge again whether it satisfies the threshold constraint. If it does, it is a moving target, otherwise it is a stationary target.

[0107] In some embodiments, calculation of the residual flow of candidate moving targets:

[0108] For the set of feature - matching points on four images calculate the object point coordinate P at time t - 1 t-1 ;

[0109] Calculate the predicted coordinate P' of object point P t-1 at time t t = RP t-1 + t;

[0110] Back - project the predicted coordinate P' t onto the image at time t to obtain the predicted image point, and the difference between this predicted image point and the observed image point is the residual optical flow ROF.

[0111] In some embodiments, judgment of the randomness of the residual flow direction:

[0112] Input: ROF of a set of candidate moving target feature points, denoted as rofs, rofs = [rof1, rof2, …, rofi, … rofn], where rofi represents the residual optical flow of the i-th feature point, and n is the number of feature points;

[0113] Output: Whether it has randomness;

[0114] Calculate the residual optical flow angle of each feature point, denoted as ang, ang = [ang1, ang2, …, angi, … angn], and the range of angi is [0 - 2π];

[0115] Sort each element in ang in ascending order;

[0116] Take the first 1 / 4 elements in ang and calculate their average value, denoted as ang_min; take the last 1 / 4 elements in ang and calculate their average value, denoted as ang_max;

[0117] Calculate the angle range ang_rang = ang_max - ang_min; when ang_rang > π, ang_rang = ang_max + (2π - ang_min);

[0118] Determine whether it meets the direction randomness; if ang_rang > thresh_ang, it is considered that the set of feature points meets the direction consistency. thresh_ang is an empirical value, recommended to be 0.8π.

[0119] In some embodiments, residual flow threshold constraint judgment:

[0120] For the set of candidate moving target feature point ROFs, first calculate the average value ave_rof of the ROF set;

[0121] Calculate the average distance of these feature points from the camera, denoted as dis_ave;

[0122] Calculate the area of the image plane where the center of these feature points is located, such as which position in 9 regions in the ROF value statistical chart, denoted as area_index;

[0123] According to dis_ave and area_index, find the corresponding threshold in the ROF value statistical chart, denoted as thresh_rof;

[0124] If ave_rof > thresh_rof, it is considered that the candidate moving target is a moving target, otherwise it is a stationary target.

[0125] Such as Figure 4As shown in the figure, the second aspect of the embodiments of this specification discloses a robust intelligent vehicle pure vision dynamic target segmentation system, including:

[0126] A field of view space ROF calibration module, which is used to set camera hardware parameters and the minimum target movement speed, perform calibration of the spatial residual flow of moving targets, and obtain the minimum residual flow in different regions of the field of view;

[0127] A calculation module, which is used to detect candidate moving targets and calculate camera self-motion parameters;

[0128] A motion state judgment module, which is used to calculate the residual flow of candidate moving targets, perform randomness judgment on the direction of the residual flow and threshold constraint judgment on the residual flow, and determine the motion state of candidate moving targets.

[0129] In some embodiments, the robust intelligent vehicle pure vision dynamic target segmentation system further includes:

[0130] A processor, which is respectively connected to the field of view space ROF calibration module, the calculation module, and the motion state judgment module;

[0131] A memory, which is connected to the processor and stores a computer program that can run on the processor;

[0132] Wherein, when the processor executes the computer program, the processor controls the field of view space ROF calibration module, the calculation module, and the motion state judgment module to work, so as to implement the above-mentioned robust intelligent vehicle pure vision dynamic target segmentation method.

[0133] The third aspect of the embodiments of this specification discloses a computer-readable storage medium. The storage medium stores computer instructions. When a computer reads the computer instructions, the computer executes the above-mentioned robust intelligent vehicle pure vision dynamic target segmentation method.

[0134] The above embodiments are used to illustrate the present invention, not to limit the present invention. Therefore, changes in the example numerical values or replacement of equivalent elements still belong to the scope of the present invention.

[0135] From the above detailed description, those of ordinary skill in the art can clearly understand that the present invention can indeed achieve the foregoing objectives, and it actually complies with the provisions of the Patent Law.

[0136] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present invention. The above description is only the preferred embodiments of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0137] It should be noted that the above description of the process is only for illustration and explanation and does not limit the scope of application of this specification. For those skilled in the art, various corrections and changes can be made to the process under the guidance of this specification. However, these corrections and changes are still within the scope of this specification.

[0138] The basic concept has been described above. Obviously, for those of ordinary skill in the art after reading this application, the above invention disclosure is only for illustration and does not constitute a limitation to this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are proposed in this application, so such modifications, improvements, and corrections still belong to the spirit and scope of the exemplary embodiments of this application.

[0139] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0140] In addition, those of ordinary skill in the art can understand that various aspects of this application can be illustrated and described by several patentable types or situations, including any new and useful process, machine, product, or combination of substances, or any new and useful improvement thereof. Therefore, various aspects of this application can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software can both be referred to as "units", "modules", or "systems". In addition, various aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, in which computer-readable program code is included.

[0141] The computer program code required for the operations of various parts of this application can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages, etc. This program code can run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (for example, through the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0142] In addition, unless clearly stated in the claims, the order of the processing elements and sequences described in this application, the use of numbers and letters, or the use of other names are not used to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details are only for illustrative purposes, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in hardware devices, it can also be implemented as a pure software solution, for example, installed on an existing server or mobile device.

[0143] Similarly, it should be noted that, in order to simplify the description of this application disclosure and thus help the understanding of one or more embodiments of the invention, in the description of the embodiments of this application above, sometimes multiple features are merged into one embodiment, drawing or description thereof. However, this method of this application should not be construed as reflecting the intention that the claimed subject matter requires more features than those clearly recited in each claim. On the contrary, the subject matter of the invention should have fewer features than the above single embodiment.

Claims

1. Robust intelligent vehicle pure vision dynamic target segmentation method, characterized by , including the following steps: S1. Perform ROF calibration; specifically: S11. Setting camera hardware parameters and target minimum motion speed; S12. Performing motion target spatial residual flow calibration; S13. Obtaining the minimum residual flow in different areas of the field of view; S2. Detect candidate moving targets and calculate camera self-motion parameters; S3. Determine the motion state of the candidate moving target; specifically: S31. Calculate the residual flow of candidate moving target feature points; S32. Perform residual flow direction randomness judgment and residual flow threshold constraint judgment to determine the motion state of the candidate moving target; S33 output motion status; The S11 includes: S111. Set the camera hardware parameters: focal length f, principal point coordinates (cx, cy), dual-camera baseline d, camera resolution H*W, where H represents the image height and W represents the image width; S112. The world coordinate system is established based on the left camera, with the camera's optical center as the world coordinate system origin. The image plane horizontally to the right is positive X, the image plane horizontally downward is positive Y, and the direction along the optical axis is positive Z. In this coordinate system, the minimum target motion parameter between two adjacent frames is: ; Respectively represent the target's movement in the X, Y, and Z directions, in meters; S113. The range of the target in the field of view refers to the front range and distance that the smart car needs to detect in the actual project. A rectangular area is designed as the area where the target is located. The rectangular area is ;Pick , the unit is m; for the rectangular area with a fixed step size Sampling is performed to obtain a set of sampling points, which are used to simulate the surface feature points of the moving target in the field of view. The three-dimensional coordinates of the surface feature points are marked as , the subscript t-1 indicates the time t-1.

2. The robust intelligent vehicle pure vision dynamic target segmentation method according to claim 1 is characterized in that: S12 includes: S121. Given target movement amount ; S122. Given the camera self-motion parameters tr=[R,T], where R is the rotation matrix of the camera at adjacent moments and T is the translation vector of the camera at adjacent moments; calculate ,Will Projected to the left image at time t Above, recorded as img_ideal_static_t; S123. Calculate the displacement of the target after movement: ; S124. Projected to the left camera at time t On the top, get the image point coordinates, recorded as img_ideal_dynamic_t; Residual optical flow ROF=img_ideal_static_t-img_ideal_dynamic_t; S125. Calculation of ROF values for object points in different regions of the image plane in different areas within the field of view: Based on the range of the target to be detected in the field of view in actual engineering, the object point distance is divided into multiple regions, divided into 6 intervals of 0-5m, 5-10m, 10-15m, 15-20m, 20-25m, and 25-50m, and the image plane is divided into multiple regions; the ROF set calculated in step S124 is statistically analyzed by distance and different regions in the image plane to obtain a ROF value statistical graph, and the minimum ROF value of each region at different distances is used as the final output value of the calibration; By changing the parameters of the camera hardware, as well as the target The values of the three components in are used to simulate the corresponding relationship between the motion state of the spatial target and its residual optical flow value in the image plane under different cameras and different scenes. The minimum value of the residual optical flow will be used as the threshold for judging the target motion state.

3. The robust intelligent vehicle pure vision dynamic target segmentation method according to claim 1 is characterized in that: In S31: right Feature matching point sets on four images , calculate the coordinates of the object point at time t-1 ; Calculate object points Predicted coordinates at time t ; The predicted coordinates Back-projected image to time t , get the predicted image point, the predicted image point and the observed image point The difference is the residual optical flow ROF.

4. The robust intelligent vehicle pure vision dynamic target segmentation method according to claim 2, characterized in that: In S32: Input: ROF of a single candidate moving target feature point set, denoted as rofs, rofs=[rof1,rof2,…,rofi,…rofn], rofi represents the residual optical flow of the i-th feature point, and n is the number of feature points; Output: whether it is random; Specifically: Calculate the residual optical flow angle of each feature point, recorded as ang, ang=[ang1,ang2,…,angi…angn], the range of angi is [0-2π]; Sort each element in ang from small to large; Take the first 1 / 4 elements in ang and calculate their average value, which is recorded as ang_min; take the last 1 / 4 elements in ang and calculate their average value, which is recorded as ang_max; Calculate the angle range ang_rang = ang_max - ang_min; When ang_rang>π, ang_rang=ang_max+(2π-ang_min); Determine whether the direction randomness is satisfied; if ang_rang>thresh_ang, the feature point set is considered to satisfy the direction consistency; thresh_ang is an empirical value.

5. The robust intelligent vehicle pure vision dynamic target segmentation method according to claim 4 is characterized in that: In S32: For the candidate moving target feature point ROF set, first calculate the average value ave_rof of the ROF set; Calculate the average distance between the candidate moving target feature points and the camera, recorded as dis_ave; Calculate the area of the image plane where the center of the candidate moving target feature point is located, recorded as area_index; According to dis_ave and area_index, find the corresponding threshold in the ROF value statistics chart, recorded as thresh_rof; If ave_rof>thresh_rof, the candidate moving target is considered to be a moving target, otherwise it is a stationary target.

6. Robust intelligent vehicle pure vision dynamic target segmentation system, characterized by: include: The field of view space ROF calibration module is used to set the camera hardware parameters and the minimum target motion speed, perform spatial residual flow calibration on the moving target, and obtain the minimum residual flow in different areas of the field of view, which is used as the threshold for motion state judgment; The calculation module is used to detect candidate moving targets and calculate the camera's self-motion parameters; The motion state judgment module is used to calculate the residual flow of the candidate moving target, perform residual flow direction randomness judgment and residual flow threshold constraint judgment, and determine the motion state of the candidate moving target; The method for setting the camera hardware parameters and the target minimum motion speed includes: Set the camera hardware parameters: focal length f, principal point coordinates (cx, cy), dual-camera baseline d, camera resolution H*W, where H represents the image height and W represents the image width; The world coordinate system is established based on the left camera, with the camera optical center as the world coordinate system origin. The image plane horizontally to the right is positive for X, the image plane horizontally downward is positive for Y, and the direction along the optical axis is positive for Z. In this coordinate system, the minimum motion parameter of the target between two adjacent frames is: ; Respectively represent the target's movement in the X, Y, and Z directions, in meters; The range of the target in the field of view refers to the front range and distance that the smart car needs to detect according to the actual engineering situation. A rectangular area is designed as the area where the target is located. ;Pick , the unit is m; for the rectangular area with a fixed step size Sampling is performed to obtain a set of sampling points, which are used to simulate the surface feature points of the moving target in the field of view. The three-dimensional coordinates of the surface feature points are marked as , the subscript t-1 indicates the time t-1.

7. The robust intelligent vehicle pure vision dynamic target segmentation system according to claim 6 is characterized in that: Also includes: A processor is connected to the field of view space ROF calibration module, the calculation module and the motion state judgment module respectively; a memory connected to the processor and storing a computer program executable on the processor; When the processor executes the computer program, the processor controls the field of view space ROF calibration module, the calculation module, and the motion state judgment module to operate, so as to implement the robust intelligent vehicle pure visual dynamic target segmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores computer instructions. When a computer reads the computer instructions, the computer executes the robust intelligent vehicle pure vision dynamic target segmentation method according to any one of claims 1 to 5.

Images