Target ranging method and device based on binocular image and medium

Abstract

The embodiment of the invention discloses a target distance measurement method and device based on a binocular image and a medium, and relates to the technical fields of target distance measurement, image matching and the like. The method comprises the following steps: taking a target frame of an object to be subjected to distance measurement in a reference image as a reference frame, and searching a first matching frame in a matching image by utilizing a template matching method; dividing the reference frame into a plurality of first sub-regions, searching a second sub-region which is most similar to each first sub-region in feature, and excluding the first sub-regions which are inconsistent in spatial distribution; determining a second matching frame of the reference frame according to the feature similarity of the remaining first sub-regions; selecting one of the first matching frame and the second matching frame, which has the maximum intersection-union ratio with the second target frame, as a final matching frame; and determining the parallax of the object in the binocular image according to the final coordinate offset of the matching frame and the reference frame, and determining the radial distance of the object according to the parallax. According to the embodiment, the accuracy and the stability of binocular distance measurement are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of target ranging, image matching, etc., and in particular to a target ranging method based on binocular images, a device and a medium. BACKGROUND

[0002] The ranging method based on target parallax map mainly uses binocular vision technology to determine the distance of a target by acquiring images through two cameras and calculating parallax. The key technical links include camera calibration and stereo matching, and the latter can calculate a parallax map through traditional algorithms such as SGBM (Semi-Global Block Matching). This method is widely used in the fields of unmanned aerial vehicles, autonomous driving, robot navigation, etc.

[0003] However, traditional algorithms such as SGBM have obvious limitations when processing targets such as unmanned aerial vehicles. For example: isolated targets have few textures, making it difficult for the algorithm to find sufficient matching feature points, and errors in parallax calculation are likely to occur; when the contrast between the target and the background is low, block matching is easily disturbed by the background, resulting in blurred edges in the parallax map; and the algorithm has poor adaptability to changes in the target's attitude, and if the unmanned aerial vehicle tilts or rotates, matching failure is likely to occur, affecting the ranging accuracy. All these reasons together reduce the accuracy and stability of target ranging. SUMMARY

[0004] Embodiments of the present application provide a target ranging method based on binocular images, a device and a medium to improve the accuracy and stability of target ranging.

[0005] In a first aspect, embodiments of the present application provide a target ranging method based on binocular images, comprising:

[0006] S10, acquiring a binocular image to be processed as a reference image and a matching image;

[0007] S20, taking a first target box of an object to be ranged in the reference image as a reference box, and using a template matching method to find a first matching box in the matching image that matches the reference box;

[0008] S30, dividing the reference box into a plurality of first sub-regions, and defining a search region in the matching image that covers a second target box of the object in the matching image, and searching for second sub-regions that are most similar in features to each first sub-region in the search region;

[0009] S40, comparing the spatial distribution of each second sub-region in the matching image with the spatial distribution of each first sub-region in the reference image, and excluding first sub-regions that are inconsistent in spatial distribution;

[0010] S50, according to the feature similarity of each remaining first sub-region and the corresponding second sub-region, performing weighted average on the coordinate offset of each remaining first sub-region and the corresponding second sub-region; and determining a second matching frame of the reference frame according to the weighted average coordinate offset;

[0011] S60, selecting one of the first matching frame and the second matching frame with the largest intersection-over-union with the second target frame as a final matching frame;

[0012] S70, determining the parallax of the object in the binocular image according to the coordinate offset of the final matching frame and the reference frame, and determining the radial distance of the object according to the parallax.

[0013] In a second aspect, an embodiment of the present application provides an electronic device, which comprises:

[0014] one or more processors;

[0015] a memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the target ranging method based on binocular images according to any embodiment.

[0017] In a third aspect, an embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target ranging method based on binocular images according to any embodiment.

[0018] In summary, the embodiment of the present application provides a target ranging method based on binocular images, which first coarsely positions a target region through cross-correlation template matching; then divides the reference frame into multiple sub-regions in a particle block manner, and matches each sub-region in the sliding window of the matching image one by one through feature similarity; after matching, the results of geometric contradictions are verified and eliminated through the spatial distribution order of the sub-regions, and the final matching frame is determined through the weighted sum intersection-over-union of the matching confidence, so as to realize accurate parallax estimation and target ranging. The whole method uses the strategy of template matching fast initialization + particle block local optimization + multi-feature cross verification, which takes into account the real-time performance and anti-interference ability, and has the following technical advantages:

[0019] 1. High precision: through the feature similarity of particle block, combined with particle order verification and double index screening, the illumination change and local occlusion are effectively overcome, and the ranging is more accurate;

[0020] 2. Strong anti-interference: through the combination of template matching initialization and particle block local optimization, the global search reliability and local matching sensitivity are taken into account;

[0021] 3. Efficiency optimization: the algorithm is lightweight, does not require training data and GPU (Graphics Processing Unit) support, and can achieve real-time processing under limited computing power, with outstanding engineering practical value.

[0022] In particular, due to the high signal interference when matching the target frame as a whole, any local area matching failure can lead to overall matching failure; therefore, the present embodiment introduces a particle block method for fine matching, so that the local image blocks are more likely to match successfully, improving the matching accuracy and stability. However, for a drone with multiple identical structures (such as the four wind wheels in a drone), the particle block matching will divide multiple sub-regions with similar structures, leading to chaotic matching between these sub-regions. Therefore, the present embodiment introduces spatial distribution verification to eliminate false matching through the spatial geometric relationship (sequential verification) between sub-regions, and determines the optimal result through matching confidence weighted average and intersection ratio coincidence degree to improve the accuracy and stability of binocular matching and target ranging. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the description of the specific embodiments or the prior art. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 is a flowchart of a target ranging method based on binocular images provided by an embodiment of the present application;

[0025] Figure 2 is a flowchart of another target ranging method based on binocular images provided by an embodiment of the present application;

[0026] Figure 3 is a flowchart of determining the final disparity according to the offset of multiple frame combinations provided by an embodiment of the present application;

[0027] Figure 4(a) is a comparison chart of the measured distance and the true distance after target ranging using the method of the present embodiment;

[0028] Figure 4(b) is a comparison chart of the measured distance and the true distance after target ranging using the SGBM algorithm;

[0029] Figure 4(c) is a comparison chart of the measured distance and the true distance after target ranging using the absolute value of the coordinate offset of the two original target frames in the binocular images as the disparity;

[0030] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0033] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0034] Figure 1 A flowchart of a target ranging method based on binocular images provided by an embodiment of the present application. The method first uses template matching method to quickly locate the target area, and at the same time uses particle block processing to accurately match, and fuses Sobel edge and RGB color features for weighted scoring matching, and then verifies and filters through multiple indexes, to improve the matching accuracy and anti-interference ability, to meet the real-time and accurate ranging requirements in complex scenes such as unmanned aerial vehicles. The method is executed by an electronic device, as shown in Figure 1 The method specifically comprises the following steps.

[0035] S110, acquiring the binocular images to be processed as a reference image and a matching image respectively.

[0036] The present embodiment first acquires binocular images at the same time, and takes any one of the monocular images therein as a reference image, and the other monocular image as a matching image, for subsequent target frame matching.

[0037] S120, taking the target frame of the object to be measured in the reference image as a reference frame, and using a template matching method to find a matching frame matching the reference frame in the matching image.

[0038] In this embodiment, the target frame of at least one target object in the two monocular images is obtained by a target detection algorithm. Optionally, the target frame of the same target object in the left monocular image can be recorded as left_box, and the target frame in the right monocular image can be recorded as right_box. This step is to preliminarily match the two target frames belonging to the same object in the two monocular images.

[0039] In a specific embodiment, the preliminary matching can include the following steps:

[0040] Step one, taking the target frame of the object to be measured in the reference image as a reference frame, and according to the reference frame and the maximum parallax, defining a search range in the matching image.

[0041] Optionally, taking the left target image as the reference image as an example, first extract the left ROI (Region of Interest). The target frame in which the object to be measured is located can be selected from the multiple target frames of the left monocular image as the ROI roi_left, which is the reference frame.

[0042] Then define the right search range. According to the two-dimensional image horizontal coordinates of the reference frame and the maximum parallax (a known quantity), a search range roi_right_search can be defined in the right monocular image, the horizontal starting coordinate of which is right_search_start, and the horizontal ending coordinate is right_search_end. The vertical coordinate interval of the search range is consistent with the vertical coordinate interval of the ROI.

[0043] Step two, using the normalized cross-correlation algorithm to perform template matching with the reference frame in the search range to obtain the best matching position. Optionally, the normalized cross-correlation algorithm (cv2.TM_CCOEFF_NORMED) can be used to perform template matching with roi_left in the search range roi_right_search to obtain the best matching position max_loc.

[0044] Step three, according to the best matching position, determine a matching frame in the matching image matching the reference frame. Specifically, moving the region in the matching image with the same coordinates as roi_left along the horizontal direction by max_loc, a matching frame in the matching image can be obtained.

[0045] The preliminary matching described above is also referred to as template matching coarse positioning in this embodiment, and the obtained matching box is referred to as a coarse positioning matching box coarse match box.

[0046] S130, divide the reference box into a plurality of sub-regions, and demarcate a search region covering the target box of the object in the matching image in the matching image, and search for another plurality of sub-regions most similar to each sub-region feature in the search region.

[0047] This step performs particle partitioning on the reference box and finds an image block matching each partition in the matching image, preparing for subsequent secondary matching of the reference box. In order to distinguish and describe, the plurality of sub-regions obtained by dividing the reference box in this embodiment are referred to as first sub-regions, and the image blocks matching each first sub-region in the matching image are referred to as second sub-regions (i.e. the "another plurality of sub-regions" described above).

[0048] In a specific embodiment, the process of the particle partitioning matching described above can include the following steps:

[0049] Step one, divide the reference box into a plurality of first sub-regions along the same direction, and complete particle initialization. Optionally, in order to take into account factors such as computing power, local information loss, and matching accuracy, the reference box can be uniformly divided into 7 particles along the horizontal direction, i.e. 7 first sub-regions.

[0050] Step two, demarcate a search region in the matching image with the target box of the object in the matching image as the center. This step determines the search center and search range in the matching image according to the object target box obtained by the target detection algorithm. Wherein, the corresponding relationship of the target boxes of the same object in the binocular images is known, in order to distinguish and describe, the target box of the same object in the reference image obtained by the target detection algorithm in S120 is referred to as the first target box, and the target box in the matching image is referred to as the second target box. Optionally, still taking the left target image as the reference image as an example, a dynamic search range search_range can be demarcated in the right eye image with the center of the right eye target box right_box as the center and expanding along the horizontal direction.

[0051] Step three, for each first sub-region, perform operations S1 to S3:

[0052] S1, determine a plurality of candidate sub-regions with the same size as the current first sub-region in the search region, and the set of these candidate sub-regions is fully covered in the search region. Optionally, slide along the horizontal direction in the search range search_range, and calculate a plurality of candidate offset amounts in turn, each candidate offset amount corresponding to a candidate sub-region in the matching image.

[0053] S2, respectively calculate the edge similarity and color similarity of each candidate sub-region and the current first sub-region, and determine the feature similarity according to the edge similarity and the color similarity.

[0054] Optionally, image preprocessing can be performed first, and the Sobel edge feature maps of the left eye image and the right eye image are calculated respectively; and the left eye image and the right eye image are converted into gray-scale images respectively, and the B (blue), G (green), and R (red) three color channels are separated.

[0055] Then, for each candidate sub-region: from the Sobel edge feature map of the reference image, the edge feature block corresponding to the current first sub-region is intercepted; from the Sobel edge feature map of the matching image, the edge feature block corresponding to the current candidate sub-region is intercepted; the difference between the two edge feature blocks is used to represent the edge similarity of the current first sub-region and the current candidate sub-region. At the same time, from the multi-channel color map of the reference image, the color feature block corresponding to the current first sub-region is intercepted; from the multi-channel color map of the matching image, the color feature block corresponding to the current candidate sub-region is intercepted; the difference between the two color feature blocks is used to represent the color similarity of the current first sub-region and the current candidate sub-region. The two kinds of differences are weighted and fused to determine the feature similarity of the current first sub-region and the current candidate sub-region. For example, the absolute error sum of the two edge feature blocks and the absolute error sum of the two color feature blocks are calculated respectively, and the two kinds of absolute error sums are weighted and fused according to the weight of 60% (edge) + 40% (color) to obtain the total score. The lower the score, the higher the feature similarity.

[0056] S3, select the candidate sub-region with the highest feature similarity as the second sub-region of the current first sub-region. Through S2, the feature similarity of each candidate sub-region and the current first sub-region can be obtained, and the candidate sub-region with the highest similarity is selected as the second sub-region.

[0057] Compared with the way of matching the target frame as a whole in S120: there are more signal disturbances in the overall matching, and any local area matching failure can lead to overall matching failure; while the particle block matching in this step focuses on local information, and the local image blocks are more likely to match successfully. Since the matching granularity after particle block is more fine, the matching in S130 of this embodiment is also called particle block fine matching.

[0058] S140, compare the spatial distribution of each second sub-region in the matching image with the spatial distribution of each first sub-region in the reference image, and exclude the first sub-region with inconsistent spatial distribution.

[0059] The step sequentially verifies the fine matching of the particle blocks. When the first sub-regions are sequentially distributed along the same direction (for example, the horizontal direction), the second sub-regions should also be sequentially distributed along the direction in the matching image, and the order of the second sub-regions should be consistent with that of the first sub-regions. If the order is not satisfied (geometric constraint), the first sub-regions and the second sub-regions that do not satisfy the order are excluded.

[0060] For example, the first sub-regions are sequentially numbered as 1, 2, 3, 4, 5, 6, and 7, and if the order of the corresponding second sub-regions is 1, 2, 5, 4, 3, 6, and 7 (the sub-regions numbered 3 and 5 do not satisfy the order constraint), the first sub-regions and the second sub-regions numbered 3 and 5 should be excluded.

[0061] S150, according to the feature similarity between the remaining first sub-regions and the corresponding second sub-regions, the coordinate offset of the remaining first sub-regions and the corresponding second sub-regions is weighted and averaged, and according to the weighted and averaged coordinate offset, another matching box of the reference frame is determined.

[0062] The step is based on the matching result of the particle blocks to perform secondary matching of the reference frame. In this embodiment, the "another matching box" obtained by the secondary matching is referred to as a refined matching box.

[0063] In a specific embodiment, for all the remaining first sub-regions, first, according to the matching confidence (i.e., feature similarity) of each first sub-region and its corresponding second sub-region, the first sub-regions are sorted in descending order, and each first sub-region is assigned a corresponding weight according to the sorting result. The higher the confidence of the first sub-region, the greater the weight proportion. Then, all the weights are normalized to ensure that the sum of the particle weights of the first sub-regions after normalization is 1. Then, the coordinate offset offset (two-dimensional image coordinate offset) of each remaining first sub-region and its corresponding second sub-region is calculated. These coordinate offsets are multiplied by the normalized weight weight of the particle, and the "offset-weight product" results of all the first sub-regions are accumulated and summed to obtain the weighted fused coordinate offset. Finally, the coordinates of the reference frame are offset according to the weighted fused coordinate offset to obtain another matching box in the matching image.

[0064] For example, still taking the left eye image as the reference image, the weighted fused offset can be denoted as refined_right_shift_candidate, and the coordinates of the reference frame are offset using the offset to obtain the horizontal coordinate refined_right_x1 of the refined matching box in the right eye image. The width and height of the right box are kept consistent with those of the reference box in the left eye image, thereby forming a refined matching box refined_right_box2.

[0065] For the convenience of distinguishing and describing, the coarse positioning matching box obtained in S120 is also referred to as a first matching box, and the refined matching box obtained in the present step is referred to as a second matching box.

[0066] S160, selecting one of the first matching box and the second matching box with the largest intersection-over-union ratio with the second target box as a final matching box.

[0067] The present step fuses and outputs the two matching boxes. Taking the left-eye image as the reference image as an example, the refined candidate box refined_right_box2 is compared with the coarse matching box coarse_match_box, and one with a higher intersection-over-union ratio with the original right_box is selected as the final matching box refined_right_box3.

[0068] Further, abnormality checking and boundary protection can be performed after the fusion. It is checked whether the absolute value of the coordinate offset of the final matching box and the second target box exceeds a set threshold value, and whether the final matching box exceeds the boundary of the matching image; if the set threshold value is exceeded or the boundary of the matching image is exceeded, the second target box is still taken as the final matching box.

[0069] For example, still taking the above left-eye image as the reference image as an example, the center of the final matching box refined_right_box3 and the center of the original right_box are checked; if the offset is too large (for example, more than 7 pixels), it is determined that the present refinement fails, and the original right_box is returned to ensure the robustness of the algorithm; at the same time, it is ensured that the output box does not exceed the image boundary. Finally, the final matching box refined_right_box is output.

[0070] S170, determining the parallax of the object in the binocular image according to the coordinate offset of the final matching box and the reference box, and determining the radial distance of the object according to the parallax.

[0071] The offset of the center coordinates of the final matching box and the reference box represents the parallax of the object in the binocular image, and the distance of the object to the binocular vision system can be calculated according to the parallax.

[0072] In a specific embodiment, in order to improve the stability of the ranging result, the reference image and the matching image in S110 to S160 can be interchanged, a new round of target box matching is performed, and the parallax information is updated according to the results of the two rounds of matching. Figure 2As shown, first, the left eye image and the right eye image in the binocular image are respectively taken as the reference image and the matching image, and the operations of S110 to S160 are performed to obtain the final matching box of the target box in the left eye image in the right eye image; also, the right eye image and the left eye image in the binocular image are respectively taken as the reference image and the matching image, and the operations of S110 to S160 are also performed to obtain the final matching box of the target box in the right eye image in the left eye image. It should be noted that the first target box and the second target box in the embodiment correspond to the reference image and the matching image respectively, and when the reference image and the matching image are transposed, the specific reference of the first target box and the second target box in the algorithm is also transposed.

[0073] Then, the target box of the object in the left eye image and the right eye image is combined with the final matching box in the left eye image and the right eye image respectively to form four square box combinations, the absolute value of the coordinate offset of the two boxes in each combination is calculated, and finally four coordinate offset absolute values are obtained. Finally, the coordinate offset absolute values corresponding to the two combinations from the binocular image are taken as candidate disparities (i.e. candidate values of the disparity), the confidence of the two candidate disparities is judged according to the coordinate offset absolute values corresponding to the two combinations from the monocular image, and the final disparity of the object in the binocular image is determined according to the judgment result.

[0074] Figure 3 is a schematic diagram provided by an embodiment of the application for determining the final disparity according to four offset absolute values. As shown, Figure 3 First, the left eye image is taken as the reference image, the target box left_box in the left eye image is taken as the reference box, and the final matching box refined_right_box corresponding to left_box is determined in the right eye image; similarly, the right eye image is taken as the reference image, the target box right_box in the right eye image is taken as the reference box, and the final matching box refined_left_box corresponding to right_box is determined in the left eye image.

[0075] Then, the target box left_box and the final matching box refined_left_box form a square box combination, the absolute value left_refinedL_x_abs of the difference value between the x coordinate (i.e. the horizontal direction coordinate) of the center point of left_box left_box_center_x and the x coordinate of the center point of refined_left_box is calculated. Since the square box combination left_box and refined_left_box are both from the left eye image, left_refinedL_x_abs belongs to the coordinate offset absolute value from the monocular image.

[0076] Similarly, a box combination is formed by the target box right_box and the final matching box refined_right_box. The absolute value of the difference between the x-coordinate of the center point of right_box and the x-coordinate of the center point of refined_right_box is calculated, and the result is right_refinedR_x_abs. Since the box combination right_box and refined_right_box are both from the right eye image, right_refinedR_x_abs is also the absolute value of the coordinate offset from the monocular image.

[0077] A box combination is formed by the target box left_box and the final matching box refined_right_box. The absolute value of the difference between the x-coordinate of the center point of left_box and the x-coordinate of the center point of refined_right_box is calculated, and the result is lr_box_x_center_diff_abs. Since the box combination left_box and refined_right_box are from the left eye image and the right eye image respectively, lr_box_x_center_diff_abs is the absolute value of the coordinate offset from the binocular image, which is a candidate disparity.

[0078] Similarly, a box combination is formed by the target box right_box and the final matching box refined_left_box. The absolute value of the difference between the x-coordinate of the center point of right_box and the x-coordinate of the center point of refined_left_box is calculated, and the result is rl_box_x_center_diff_abs. Since the box combination right_box and refined_left_box are from the right eye image and the left eye image respectively, rl_box_x_center_diff_abs is also the absolute value of the coordinate offset from the binocular image, which is another candidate disparity.

[0079] After obtaining the above four coordinate offset absolute values, the two coordinate offset absolute values from the monocular image are taken as the confidence of the two candidate disparities respectively. Specifically, the coordinate offset absolute value left_refinedL_x_abs from the left eye image is taken as the confidence of the candidate disparity rl_box_x_center_diff_abs calculated according to the final matching box refined_left_box in the left eye image. The smaller the coordinate offset left_refinedL_x_abs from the left eye image, the more reasonable the final matching box refined_left_box in the left eye image is matched, and the higher the confidence of the candidate disparity rl_box_x_center_diff_abs calculated by refined_left_box is. Similarly, the coordinate offset absolute value right_refinedR_x_abs from the right eye image is taken as the confidence of the candidate disparity lr_box_x_center_diff_abs calculated according to the final matching box refined_right_box in the right eye image. The smaller the coordinate offset right_refinedR_x_abs from the right eye image, the more reasonable the final matching box refined_right_box in the right eye image is matched, and the higher the confidence of the candidate disparity lr_box_x_center_diff_abs calculated by refined_right_box is.

[0080] Based on the above representative relationship, combined with Figure 3 , when determining the disparity, first determine whether left_refinedL_x_abs<5 and right_refinedR_x_abs<5 are satisfied, that is, whether the confidence of the two candidate disparities is high enough.

[0081] If yes (satisfying left_refinedL_x_abs<5 and right_refinedR_x_abs<5), both candidate disparities have high confidence, and the absolute value bi_disparity_diff_abs of the difference value of the two candidate disparities is calculated. If the absolute value is <3, it can be confirmed again that the difference between the two candidate disparities is within a reasonable range and has high confidence, and the two candidate disparities are taken as the final disparity. If the absolute value is ≥3, the size of left_refinedL_x_abs and right_refinedR_x_abs is compared, and the candidate disparity with higher confidence is taken as the final disparity.

[0082] If no (not satisfying "left_refinedL_x_abs<5 and right_refinedR_x_abs<5"), continue to judge whether left_refinedL_x_abs<5 or right_refinedR_x_abs<5, i.e. whether there is any one of the two candidate boxes with high enough confidence. If yes (satisfying left_refinedL_x_abs<5 or right_refinedR_x_abs<5), continue to compare the size of left_refinedL_x_abs and right_refinedR_x_abs, and take the candidate with higher confidence as the final disparity. If no (left_refinedL_x_abs≥5 and right_refinedR_x_abs≥5), return to the two target boxes, and take the coordinate offset amount box_x_center_disparity of the two target boxes as the final disparity.

[0083] It should be noted that the threshold values 5 and 3 in the above judgment represent 5 pixels and 3 pixels respectively, and the values of each threshold can be set as needed. If the three pixel threshold values 7 pixels, 5 pixels and 3 pixels appearing in the above are denoted as a first threshold value, a second threshold value and a third threshold value respectively, it is preferred to satisfy the values of the first threshold value>the second threshold value>the third threshold value, such as 7>5>3.

[0084] After obtaining the final disparity, the three-dimensional coordinates (three-dimensional coordinates in the camera coordinate system) of the target object are calculated, and the distance information of the target can be obtained. Specifically, given the binocular disparity d (the horizontal displacement difference of the object in the image plane), the binocular camera intrinsic parameters and the baseline (the horizontal distance between the binocular cameras), the distance of the target can be calculated through the following steps:

[0085] First, the depth of the object is calculated according to the disparity. The disparity is the horizontal difference between the corresponding points in the two images, which reflects the distance between the object and the camera. The relationship between the disparity and the object depth can be described by the following formula:

[0086]

[0087] Where Z represents the object depth, i.e. the Z coordinate in the camera coordinate system; f represents the focal length of the camera, which is usually provided by the intrinsic parameters; B represents the baseline of the binocular camera, i.e. the horizontal distance between the two cameras; d represents the disparity value, i.e. the horizontal displacement difference of the same object in the two images.

[0088] Then, the three-dimensional coordinates in the object camera coordinate system are calculated. For each pixel point, assuming its two-dimensional image coordinates are (x, y), the three-dimensional coordinates in the camera coordinate system can be calculated by the following formula:

[0089]

[0090]

[0091] wherein X and Y represent the coordinates of each pixel in the object in the camera coordinate system, corresponding to the width and height of the object respectively, is the position of the camera principal point (i.e. the intersection of the optical axis and the image plane) in the image.

[0092] Finally, the radial distance is calculated. The radial distance is the straight-line distance between the object and the origin of the camera coordinate system, which can be calculated from the three-dimensional coordinates:

[0093]

[0094] wherein R is the radial distance between the object and the origin of the camera coordinate system, i.e. the final ranging result of the embodiment.

[0095] Figures 4(a) to 4(c) is a comparison chart of the target ranging results using the method of the embodiment and the other two methods. The three charts all correspond to the same binocular image sequence, wherein Fig. 4(a) is a comparison chart of the measured distance and the true distance after the target ranging is performed frame by frame using the method of the embodiment; Fig. 4(b) is a comparison chart of the measured distance and the true distance after the target ranging is performed frame by frame using the SGBM algorithm; Fig. 4(c) is a comparison chart of the measured distance and the true distance after the target ranging is performed by taking the absolute value of the coordinate offset of the two original target boxes in the binocular image as the disparity. It can be seen that the method of the embodiment has obvious improvement in ranging accuracy and stability.

[0096] In summary, the embodiment provides a target ranging method based on binocular images. First, the target region is roughly positioned by normalized cross-correlation template matching; then, the reference box is divided into multiple sub-regions by particle block method, each sub-region is matched one by one in the sliding window of the matching image by weighted scoring of Sobel edge features (such as weight 60%) and RGB three-channel color features (such as weight 40%); after matching, the results of geometric contradiction are eliminated by verifying the spatial distribution order of the sub-regions, and the optimal offset is determined by the weighted sum intersection ratio coincidence degree of the matching confidence; finally, the high-precision final matching box is output by dynamic boundary protection, realizing accurate disparity estimation and target ranging. The whole method uses the strategy of template matching fast initialization + particle block local optimization + multi-feature cross verification, which takes into account the real-time performance and anti-interference ability, and has the following technical advantages:

[0097] 1. High precision: by Sobel edge and RGB color weighted fusion, combined with particle sequential verification and double index screening, effectively overcoming illumination changes and local occlusion, more accurate ranging;

[0098] 2. Strong anti-interference: combined with template matching initialization and particle block local optimization, taking into account the reliability of global search and the sensitivity of local matching; through the pairwise combination of binocular original target box and binocular final matching box, the binocular time distance is determined comprehensively, and the ranging stability is further improved.

[0099] 3. Efficiency: the algorithm is lightweight, does not require training data and GPU support, and can achieve real-time processing under limited computing power, with outstanding engineering practical value. Compared with the method of the application, the traditional global optimization stereo matching algorithm constructs a cost cube through mutual information or Census transformation, and performs cost aggregation, which can obtain a dense and high-precision disparity map, but the calculation amount is large; the deep learning method uses an end-to-end deep neural network to directly regress a disparity map through feature extraction, cost construction and 3D convolution regularization, which requires a large amount of data training and GPU computing power.

[0100] In particular, for the difficulty of ranging for a target object such as a drone which has a symmetrical structure and multiple identical structures, the present embodiment proposes multiple improvement measures: first, since there is more signal interference when matching the target box as a whole, any local area matching failure can lead to overall matching failure; therefore, the present embodiment introduces a particle block method for fine matching, making it easier to match between local image blocks and improving matching accuracy and stability. However, for an object such as a drone which has multiple identical structures (such as the four wind wheels in a drone), particle block matching will divide multiple sub-regions with similar structures, leading to matching confusion between these sub-regions, so the present embodiment introduces spatial distribution verification to eliminate false matches through spatial geometric relationships between sub-regions (sequential verification), and determines the optimal result through matching confidence weighted average and intersection-over-union overlap. In addition, the present embodiment also performs multi-feature weighted scoring and fusion, using Sobel edge features (weight 60%) and RGB color features (weight 40%) to improve matching accuracy; a dynamic boundary protection mechanism is set to prevent the final matching box from exceeding the boundary when output; and the pairwise combination of binocular original target box and binocular final matching box is used to determine the binocular time distance comprehensively. These measures are all anti-interference measures specially set for a target object such as a drone which has a symmetrical structure and multiple identical structures, aiming to improve the accuracy and stability of binocular matching and target ranging.

[0101] It should be noted that all data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0102] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 5 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0103] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the target ranging method based on binocular images in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby implementing the aforementioned target ranging method based on binocular images.

[0104] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.

[0106] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target ranging method based on binocular images of any embodiment.

[0107] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0108] The computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can be used to carry or propagate program code that is used by or in connection with an instruction execution system, apparatus or device.

[0109] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0110] The computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A method for target ranging based on binocular images, characterized in that, The method comprises the following steps: S10, acquiring a to-be-processed binocular image as a reference image and a matching image respectively; S20, taking a first target frame of an object to be measured in the reference image as a reference frame, and searching for a first matching frame matching the reference frame in the matching image by using a template matching method; S30, dividing the reference frame into a plurality of first sub-regions, and demarcating a search region covering a second target frame of the object in the matching image in the matching image, and searching for a second sub-region most similar to each first sub-region in the search region respectively; S40, comparing the spatial distribution of each second sub-region in the matching image with the spatial distribution of each first sub-region in the reference image, and excluding the first sub-region with inconsistent spatial distribution; S50, according to the feature similarity between the remaining first sub-regions and the corresponding second sub-regions, performing a weighted average on the coordinate offset of the remaining first sub-regions and the corresponding second sub-regions; and determining the second matching frame of the reference frame according to the weighted average coordinate offset; S60, selecting one of the first matching frame and the second matching frame with the largest intersection-over-union with the second target frame as the final matching frame; S70, determining the parallax of the object in the binocular image according to the coordinate offset between the final matching frame and the reference frame, and determining the radial distance of the object according to the parallax.

2. The method of claim 1, wherein, The object is an object with a plurality of identical structures; The object includes a UAV, and the plurality of identical structures of the UAV include a plurality of wind wheels.

3. The method of claim 1, wherein, The method of searching for a first matching frame matching the reference frame in the matching image by using a template matching method comprises the following steps: Demarcating a search range in the matching image according to the reference frame and the maximum parallax; In the search range, template matching is performed with the reference frame by using a normalized cross-correlation algorithm to obtain an optimal matching position; According to the optimal matching position, a first matching frame matching the reference frame in the matching image is determined.

4. The method of claim 1, wherein, The method of dividing the reference frame into a plurality of first sub-regions comprises the following steps: The reference frame is evenly divided into a plurality of first sub-regions along the same direction; 5. The method of claim 1, wherein, Correspondingly, S40 comprises the following steps: Comparing the order of each second sub-region in the matching image along the same direction with the order of each first sub-region in the reference image along the same direction, and excluding the first sub-region with inconsistent order. The method of demarcating a search region covering the second target frame of the object in the matching image in the matching image, and searching for a second sub-region most similar to each first sub-region in the search region comprises the following steps: Demarcating a search region in the matching image with the second target frame of the object in the matching image as the center; For each first sub-region: a plurality of candidate sub-regions with the same size as the current first sub-region and covering the search region are determined in the search region, the edge similarity and color similarity between each candidate sub-region and the current first sub-region are calculated respectively, the feature similarity is determined according to the edge similarity and color similarity, and the candidate sub-region with the highest feature similarity is selected as the second sub-region.

6. The method of claim 5, wherein, The calculating the edge similarity and the color similarity of each candidate sub-region and the current first sub-region respectively, and determining the feature similarity according to the edge similarity and the color similarity, comprises: cutting an edge feature block corresponding to the current first sub-region from a Sobel edge feature map of the reference image, and cutting an edge feature block corresponding to the current candidate sub-region from a Sobel edge feature map of the matching image; and using the difference between the two edge feature blocks to represent the edge similarity between the current first sub-region and the current candidate sub-region; cutting a color feature block corresponding to the current first sub-region from a multi-channel color map of the reference image, and cutting a color feature block corresponding to the current candidate sub-region from a multi-channel color map of the matching image; and using the difference between the two color feature blocks to represent the color similarity between the current first sub-region and the current candidate sub-region; weighting and fusing the two differences to determine the feature similarity between the current first sub-region and the current candidate sub-region.

7. The method of claim 1, wherein, After S60, further comprising: checking whether the absolute value of the coordinate offset of the final matching frame and the second target frame exceeds a set threshold, and whether the final matching frame exceeds the boundary of the matching image; if the set threshold is exceeded or the boundary of the matching image is exceeded, still taking the second target frame as the final matching frame.

8. The method of claim 1, wherein, Before S70, further comprising: taking the left image and the right image in the binocular image as the reference image and the matching image respectively, performing the operations of S10 to S60 to obtain the final matching frame in the right image; taking the right image and the left image in the binocular image as the reference image and the matching image respectively, performing the operations of S10 to S60 to obtain the final matching frame in the left image; Correspondingly, the determining the disparity of the object in the binocular image according to the coordinate offset of the final matching frame and the reference frame comprises: combining the target frame of the object in the left image and the right image with the final matching frame in the left image and the right image respectively to calculate the absolute value of the coordinate offset; taking the absolute value of the coordinate offset corresponding to the two combinations from the binocular image as a candidate disparity, judging the confidence of the two candidate disparities according to the absolute value of the coordinate offset corresponding to the two combinations from the monocular image, and determining the final disparity of the object in the binocular image according to the judgment result.

9. An electronic device, comprising: comprise: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the binocular image-based target ranging method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, a computer program is stored thereon, which is executed by a processor to implement the binocular image-based target ranging method of any one of claims 1-8.

Images