Binocular vision positioning method and device based on deep learning, equipment, medium and product
By using a deep learning inference engine on terminal devices to process binocular vision sensor data, calculate parallax values and physical coordinates of obstacles, the impact of environmental transformation on binocular vision positioning is solved, and obstacle recognition and positioning is achieved with higher accuracy.
Patent Information
- Application Number
- CN202411281398.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2025-08-12
AI Technical Summary
Existing binocular visual positioning methods are susceptible to environmental changes, resulting in low accuracy of positioning results.
By setting up a binocular vision sensor on the terminal device, using a pre-constructed deep learning-based inference engine to process the left and right obstacle data, calculate the disparity value set, and combine sensor parameters to calculate the depth value and physical coordinates of the obstacle to achieve accurate obstacle positioning.
It improves the accuracy of binocular visual positioning results, reduces the impact of environmental transformation on identifying obstacles, and improves obstacle recognition efficiency and positioning accuracy.
Smart Images
Figure CN120471992A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a binocular vision positioning method, device, equipment, medium and product based on deep learning. Background Art
[0002] Binocular vision navigation is a key technology in the field of robotic visual navigation. By simulating the human binocular vision system, it acquires spatial depth information, enabling navigation and obstacle avoidance. In recent years, with the development of computer vision and artificial intelligence technologies, binocular vision navigation technology based on deep learning has made significant progress.
[0003] To achieve robot navigation and obstacle avoidance using binocular vision navigation technology, it is first necessary to accurately locate the robot using binocular vision positioning technology. Currently, binocular vision positioning for robots typically relies on stereo matching technology or disparity estimation algorithms. However, in practice, existing binocular vision positioning methods have been found to be susceptible to environmental changes, resulting in low accuracy in binocular vision positioning results. Summary of the Invention
[0004] The purpose of this application is to provide a binocular vision positioning method, device, equipment, medium and product based on deep learning, which can improve the accuracy of binocular vision positioning results.
[0005] To achieve the above objectives, this application provides the following solutions:
[0006] In a first aspect, the present application provides a binocular vision positioning method based on deep learning, the method being applied to a terminal device, the terminal device being provided with a binocular vision sensor, the binocular vision sensor including a left vision sensor and a right vision sensor, the binocular vision positioning method based on deep learning comprising:
[0007] Collecting left-side obstacle data and right-side obstacle data using the binocular vision sensor; wherein the left-side obstacle data includes a left-side image, which includes an image of the first obstacle and an image of the second obstacle; and the right-side obstacle data includes a right-side image, which includes an image of the first obstacle and an image of the second obstacle;
[0008] Processing the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle and second left detection frame information and second right detection frame information of the second obstacle; wherein the inference engine is built based on a deep learning model;
[0009] Obtaining a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information;
[0010] Calculating a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor;
[0011] Calculate the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters;
[0012] The bottom coordinates are converted into physical coordinates of an obstacle based on the sensor parameters and the depth value.
[0013] Optionally, the method for constructing the inference engine includes:
[0014] The constructed deep learning model is trained based on the acquired obstacle dataset to obtain a trained deep learning model;
[0015] Obtaining a weight file of the deep learning model; wherein the weight file includes parameters and weight data of the deep learning model;
[0016] Convert the weight file into a wst file;
[0017] The wst file is sent to the terminal device, and the wst file is converted into an engine file on the terminal device, so that the terminal device builds an inference engine based on the engine file.
[0018] Optionally, obtaining a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information specifically includes:
[0019] Performing edge detection on the first left detection frame information and the first right detection frame information to obtain a first contour line segment of the first obstacle and a second contour line segment of the second obstacle;
[0020] determining a first edge segment of the first obstacle from the first contour segment;
[0021] determining a second edge segment of the second obstacle from the second contour segment;
[0022] Performing edge detection on the second left detection frame information to obtain a third edge segment corresponding to the first edge segment;
[0023] Performing edge detection on the second right detection frame information to obtain a fourth edge segment corresponding to the second edge segment;
[0024] Acquire a plurality of first feature points from the first edge segment, and acquire a third feature point corresponding to the first feature point from the third edge segment;
[0025] Calculating a first disparity value between each of the first feature points and a third feature point corresponding to the first feature point;
[0026] Acquire multiple second feature points from the second edge segment, and acquire fourth feature points corresponding to the second feature points from the fourth edge segment;
[0027] calculating a second disparity value between each second feature point and a fourth feature point corresponding to the second feature point;
[0028] A plurality of first disparity values and a plurality of second disparity values are added to a set of disparity values between the left image and the right image.
[0029] Optionally, the calculating, based on the disparity value set and sensor parameters of the binocular vision sensor, a depth value from the first obstacle to the optical center of the left vision sensor specifically includes:
[0030] Obtaining a baseline length, a focal length of the left vision sensor, a left view principal point physical coordinate of a left view principal point of the left vision sensor, and a right view principal point physical coordinate of a right view principal point of the right vision sensor from sensor parameters of the binocular vision sensor; wherein the baseline length is the distance between the left vision sensor and the right vision sensor;
[0031] Calculating each disparity value in the disparity value set based on the baseline length, the focal length, the physical coordinates of the left view principal point, and the physical coordinates of the right view principal point to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor; wherein the initial depth value set includes a plurality of initial depth values;
[0032] Determining the number of depth values of the initial depth value included in the initial depth value set;
[0033] Based on the number of depth values, an average of the initial depth values in the depth value set is calculated to obtain a depth value from the first obstacle to the optical center of the left vision sensor.
[0034] Optionally, calculating the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters specifically includes:
[0035] Obtain a first line segment parameter of a first expression of the first edge line segment;
[0036] Obtain a second line segment parameter of a second expression of the second edge line segment;
[0037] collecting the current distance between the first obstacle and the second obstacle by a preset distance measuring sensor;
[0038] Calculate the bottom vertical coordinate of the bottom feature point of the first obstacle in the left image based on the first line segment parameter, the second line segment parameter, the current distance, the depth value, and the focal length;
[0039] Calculate the bottom horizontal coordinate of the bottom feature point based on the bottom vertical coordinate and the first expression;
[0040] The bottom horizontal coordinate and the bottom vertical coordinate are combined to obtain the bottom coordinate of the bottom feature point.
[0041] Optionally, converting the bottom coordinates into physical coordinates of an obstacle based on the sensor parameters and the depth value specifically includes:
[0042] Determine the center point coordinates of the left image;
[0043] Decentralize the bottom coordinates based on the center point coordinates to obtain target bottom coordinates;
[0044] The target bottom coordinates are converted into obstacle physical coordinates based on the depth value and the focal length.
[0045] In a second aspect, the present application provides a binocular vision positioning device based on deep learning, the device being applied to a terminal device, the terminal device being provided with a binocular vision sensor, the binocular vision sensor including a left vision sensor and a right vision sensor, the binocular vision positioning device based on deep learning comprising:
[0046] a collection unit configured to collect left-side obstacle data and right-side obstacle data using the binocular vision sensor; wherein the left-side obstacle data includes a left-side image, which includes an image of the first obstacle and an image of the second obstacle; and the right-side obstacle data includes a right-side image, which includes an image of the first obstacle and an image of the second obstacle.
[0047] a first processing unit, configured to process the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle, and second left detection frame information and second right detection frame information of the second obstacle; wherein the inference engine is built based on a deep learning model;
[0048] a second processing unit, configured to obtain a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information;
[0049] a first calculating unit, configured to calculate a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor;
[0050] a second calculation unit, configured to calculate a bottom coordinate of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters;
[0051] A conversion unit is configured to convert the bottom coordinates into physical coordinates of an obstacle based on the sensor parameters and the depth value.
[0052] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned binocular visual positioning methods based on deep learning.
[0053] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-mentioned binocular vision positioning methods based on deep learning.
[0054] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned binocular vision positioning methods based on deep learning.
[0055] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0056] The present application provides a binocular vision positioning method, apparatus, equipment, medium and product based on deep learning. The binocular vision sensor collects left-side obstacle data including a left-side image and right-side obstacle data including a right-side image, and both the left-side image and the right-side image contain a first obstacle image and a second obstacle image. Based on the obtained left-side image and right-side image, the detection frames of the first obstacle and the second obstacle in the left image and the right image can be determined, thereby solving the influence of the environment on the recognition of obstacles, so that the binocular vision positioning method provided by the present application can be unaffected by environmental changes and can more accurately identify the detection frame of the obstacle. The information of the detection frame can then be processed to determine the position information of the obstacle relative to the terminal device, thereby improving the accuracy of the binocular vision positioning results. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0058] Figure 1 This is a flowchart of a binocular vision positioning method based on deep learning in one embodiment of the present application;
[0059] Figure 2 A schematic diagram of a detection frame of a first obstacle and a second obstacle provided in one embodiment of the present application;
[0060] Figure 3 A schematic diagram of an image coordinate system and a physical coordinate system provided in one embodiment of the present application;
[0061] Figure 4 A schematic diagram of a module of a binocular vision positioning device based on deep learning provided in one embodiment of the present application;
[0062] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0064] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0065] In the embodiments of the present application, terminal devices may include, but are not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, smart sweeping robots, smart AGVs, etc. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc.
[0066] In an exemplary embodiment, Figure 1 As shown, a binocular vision positioning method based on deep learning is provided, which is applied to a terminal device. The terminal device is provided with a binocular vision sensor, and the binocular vision sensor includes a left vision sensor and a right vision sensor. In an embodiment of the present application, the method includes the following steps 101 to 106. Among them:
[0067] Step 101: Collect left obstacle data and right obstacle data through the binocular vision sensor.
[0068] In an embodiment of the present application, the left obstacle data includes a left image, which includes an image of the first obstacle and an image of the second obstacle; the right obstacle data includes a right image, which includes an image of the first obstacle and an image of the second obstacle.
[0069] Please also refer to Figure 2 , Figure 2 A schematic diagram of a detection frame of a first obstacle and a second obstacle provided in an embodiment of the present application is shown. Figure 2 The image shown may include an image of the first obstacle and a left image or a right image of the second obstacle, and the quadrilateral object on the left side of the image is the first obstacle, and the object on the right side is the second obstacle.
[0070] Step 102: Process the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle and second left detection frame information and second right detection frame information of the second obstacle.
[0071] In an embodiment of the present application, the inference engine is built based on a deep learning model. The deep learning model can identify a first obstacle and a second obstacle from the left obstacle data and the right obstacle data, and further determine first left detection frame information and first right detection frame information of the first obstacle, and second left detection frame information and second right detection frame information of the second obstacle.
[0072] Therefore, the functions of the trained deep learning model can be transferred to the terminal device. The specific transfer method is to build an inference engine on the terminal device based on the model parameters of the deep model.
[0073] As an optional implementation, a method for constructing an inference engine includes:
[0074] The constructed deep learning model is trained based on the acquired obstacle dataset to obtain a trained deep learning model;
[0075] Obtaining a weight file of the deep learning model; wherein the weight file includes parameters and weight data of the deep learning model;
[0076] Convert the weight file into a wst file;
[0077] The wst file is sent to the terminal device, and the wst file is converted into an engine file on the terminal device, so that the terminal device builds an inference engine based on the engine file.
[0078] Among them, by implementing this implementation method, the deep learning model can be trained according to the obtained obstacle data set to obtain a deep learning model with accurate recognition results; and the weight file in the deep learning model can be converted into a wst file that can be applied by the terminal device, so that the terminal device can build an inference engine based on the wst file, so that the terminal device can accurately identify the relevant information of the obstacle based on the inference engine, without sending the collected obstacle image to the service device for identification, thereby improving the efficiency of obstacle identification.
[0079] In the embodiment of the present application, the format of the weight file of the deep learning model can be pth format, and the parameters and weight data of the deep learning model can be parameters such as the weight matrix and bias vector of each layer in the deep learning model. The wst file can be used to transfer model parameters between different frameworks. The wst file can contain layer names and corresponding weight values.
[0080] Optionally, the terminal device may construct an inference engine based on the wst file in the following manner:
[0081] The terminal device can convert the wts file into the engine file used for terminal device reasoning. The engine file is an optimized model file generated by TensorRT. The engine file can contain the optimized calculation graph, weights and parameters, and then build an inference engine based on the engine file.
[0082] In the embodiment of the present application, the detection frame information may include information such as obstacle type, obstacle confidence, obstacle image coordinates, and obstacle length and width.
[0083] For example, the first left detection frame information of the first obstacle may include information such as the obstacle type, obstacle confidence, obstacle image coordinates, and obstacle length and width of the first obstacle identified in the left image.
[0084] In an embodiment of the present application, the obstacle type may be a stationary obstacle type or a moving obstacle type, the obstacle confidence may be the credibility of the obstacle type, the obstacle image coordinates may be the coordinates in the left image coordinate system corresponding to the left image, and information such as the obstacle length and width may also be parameters based on the left image coordinate system corresponding to the left image.
[0085] Similarly, the first right detection frame information of the first obstacle and the second left detection frame information and second right detection frame information of the second obstacle can be obtained.
[0086] from Figure 2 It can be seen that the rectangular boxes surrounding the first obstacle and the second obstacle are the first left detection box information or the first right detection box information of the first obstacle and the second left detection box information or the second right detection box information of the second obstacle.
[0087] Step 103 : Obtain a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information.
[0088] In the embodiment of the present application, the left image corresponds to the left image coordinate system, and the right image corresponds to the right image coordinate system. This indicates that the coordinate systems of images captured by different visual sensors are different, meaning that there is parallax between the left image coordinate system and the right image coordinate system, and thus, between the left and right images. Therefore, it is necessary to determine the parallax between the left and right images based on the identified first left detection frame information, first right detection frame information, second left detection frame information, and second right detection frame information.
[0089] To determine the disparity between the left and right images, it is necessary to obtain matching feature points in the left and right images, and then obtain the disparity value between the corresponding feature points. To ensure the accuracy of the disparity calculation, it is necessary to select multiple pairs of feature points, and then obtain the disparity value of each pair of feature points, and then obtain a disparity value set containing the disparity value of each pair of feature points. Based on the disparity values of the multiple pairs of feature points contained in the disparity value set, a more accurate disparity value can be obtained.
[0090] As an optional implementation manner, step 103 of obtaining a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information may specifically include:
[0091] Performing edge detection on the first left detection frame information and the first right detection frame information to obtain a first contour line segment of the first obstacle and a second contour line segment of the second obstacle;
[0092] determining a first edge segment of the first obstacle from the first contour segment;
[0093] determining a second edge segment of the second obstacle from the second contour segment;
[0094] Performing edge detection on the second left detection frame information to obtain a third edge segment corresponding to the first edge segment;
[0095] Performing edge detection on the second right detection frame information to obtain a fourth edge segment corresponding to the second edge segment;
[0096] Acquire a plurality of first feature points from the first edge segment, and acquire a third feature point corresponding to the first feature point from the third edge segment;
[0097] Calculating a first disparity value between each of the first feature points and a third feature point corresponding to the first feature point;
[0098] Acquire multiple second feature points from the second edge segment, and acquire fourth feature points corresponding to the second feature points from the fourth edge segment;
[0099] calculating a second disparity value between each second feature point and a fourth feature point corresponding to the second feature point;
[0100] A plurality of first disparity values and a plurality of second disparity values are added to a set of disparity values between the left image and the right image.
[0101] Among them, when implementing this embodiment, edge detection can be first performed on the first left detection frame information and the first right detection frame information to determine the first edge segment of the first obstacle and the second edge segment of the second obstacle, and the third edge segment corresponding to the first edge segment and the fourth edge segment corresponding to the second edge segment can be determined from the second left detection frame information and the second right detection frame information; and then, a disparity value set can be calculated based on the position corresponding feature points obtained from the first edge segment and the third edge segment, and based on the position corresponding feature points obtained from the second edge segment and the fourth edge segment, thereby improving the accuracy of the disparity values contained in the disparity value set.
[0102] In the embodiment of the present application, edge detection can be implemented using the Hough line detection algorithm. These contour segments are represented by the x and y coordinates of the two endpoints. Each segment can be represented as a coordinate pair of two points: (x1, y1) and (x2, y2) representing the starting point and end point of the contour segment, respectively. There can be multiple first contour segments and second contour segments.
[0103] Furthermore, the first contour line segment and the second contour line segment can be screened, the slopes and center points of the first contour line segment and the second contour line segment can be obtained, and the first contour line segment whose slope is in the preset slope range and the center point of the line segment is in the first preset area in the left image can be determined as the first edge line segment, and the second contour line segment whose slope is in the preset slope range and the center point of the line segment is in the second preset area in the left image can be determined as the second edge line segment, wherein the first preset area and the second preset area can be the same or different.
[0104] Furthermore, based on the first edge segment of the first obstacle determined from the left image, the third edge segment of the first obstacle in the right image that matches the position of the first edge segment can be determined; and based on the second edge segment of the first obstacle determined from the left image, the fourth edge segment of the second obstacle in the right image that matches the position of the second edge segment can also be determined.
[0105] Optionally, the expression of the first edge segment may be:
[0106] y1=k1x1+b1
[0107] Among them, k1 and b1 are the parameters of the first line segment.
[0108] The expression of the second edge segment can be:
[0109] y2=k2x2+b2
[0110] Among them, k2 and b2 are the parameters of the second line segment.
[0111] See also Figure 2 , Figure 2 The outermost edge contour of the first obstacle in may represent the first edge segment of the first obstacle or the third edge segment of the first obstacle, Figure 2 The outermost edge contour of the second obstacle in may represent the second edge segment of the second obstacle or the fourth edge segment of the second obstacle.
[0112] Step 104 : Calculate a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor.
[0113] As an optional implementation manner, step 104 may specifically include calculating the depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and the sensor parameters of the binocular vision sensor:
[0114] Obtaining a baseline length, a focal length of the left vision sensor, a left view principal point physical coordinate of a left view principal point of the left vision sensor, and a right view principal point physical coordinate of a right view principal point of the right vision sensor from sensor parameters of the binocular vision sensor; wherein the baseline length is the distance between the left vision sensor and the right vision sensor;
[0115] Calculating each disparity value in the disparity value set based on the baseline length, the focal length, the physical coordinates of the left view principal point, and the physical coordinates of the right view principal point to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor; wherein the initial depth value set includes a plurality of initial depth values;
[0116] Determining the number of depth values of the initial depth value included in the initial depth value set;
[0117] Based on the number of depth values, an average of the initial depth values in the depth value set is calculated to obtain a depth value from the first obstacle to the optical center of the left vision sensor.
[0118] Among them, by implementing this embodiment, each disparity value in the disparity value set can be calculated based on the acquired sensor parameters of the binocular vision sensor, together with the physical coordinates of the principal point of the left view and the physical coordinates of the principal point of the right view, to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor, and then the average value of the initial depth value set can be determined as the depth value from the first obstacle to the optical center of the left vision sensor, thereby improving the accuracy of the depth value calculation.
[0119] In the embodiment of the present application, the calculation formula for the initial depth value D from the first obstacle to the optical center of the left vision sensor can be:
[0120]
[0121] Wherein, B represents the baseline length, f represents the focal length, d represents the parallax value, Xr represents the physical coordinates of the principal point of the right view, and Xl represents the physical coordinates of the principal point of the left view.
[0122] Then, based on the calculated initial depth value set, an average value of the initial depth value set, ie, the depth value Z from the first obstacle to the optical center of the left vision sensor, can be further calculated.
[0123] Step 105 : Calculate the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters.
[0124] In the embodiments of this application, the physical coordinates of the principal point of the left view, the physical coordinates of the principal point of the right view, and the disparity value are all referenced to the left image coordinate system. The baseline length and focal length are both referenced to real physical parameters, that is, they can be regarded as being based on the real physical coordinate system.
[0125] Therefore, there is a conversion relationship between the image coordinate system on the left and the real physical coordinate system. Please refer to Figure 3 , Figure 3 This is a schematic diagram of an image coordinate system and a physical coordinate system provided in one embodiment of the present application. The optical center of the left visual sensor is the coordinate origin Oc of the real physical coordinate system, the value of |O'Oc| is the focal length f, the Xc and Yc axes of the real physical coordinate system are parallel to the X and Y axes of the left image coordinate system, and the Zc axis is perpendicular to the image plane. Figure 3 As shown, let the homogeneous coordinates of point P in the real physical coordinate system be (Xc, Yc, Zc, 1), and the homogeneous coordinates of its corresponding image point in the left image coordinate system be (x, y, 0, 1). Figure 3 The geometric relationship shown in the following formula can be obtained:
[0126]
[0127] Then we can deduce:
[0128]
[0129] As an optional implementation manner, step 105 may specifically include calculating the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters:
[0130] Obtain a first line segment parameter of a first expression of the first edge line segment;
[0131] Obtain a second line segment parameter of a second expression of the second edge line segment;
[0132] collecting the current distance between the first obstacle and the second obstacle by a preset distance measuring sensor;
[0133] Calculate the bottom vertical coordinate of the bottom feature point of the first obstacle in the left image based on the first line segment parameter, the second line segment parameter, the current distance, the depth value, and the focal length;
[0134] Calculate the bottom horizontal coordinate of the bottom feature point based on the bottom vertical coordinate and the first expression;
[0135] The bottom horizontal coordinate and the bottom vertical coordinate are combined to obtain the bottom coordinate of the bottom feature point.
[0136] Among them, by implementing this embodiment, calculations can be performed based on the parameters of the first expression of the first edge segment, the parameters of the second expression of the second edge segment, the current distance, depth value and focal length collected between the first obstacle and the second obstacle, and the horizontal and vertical coordinates of the bottom feature point of the first obstacle in the left image can be obtained, thereby improving the accuracy of the coordinate calculation of the bottom feature point.
[0137] Therefore, the relationship between the image coordinate system on the left and the real physical coordinate system is:
[0138]
[0139] Where zd is the current distance between the first obstacle and the second obstacle, and Δx is the horizontal distance in the x-direction between the bottom feature point of the first obstacle and the bottom feature point of the second obstacle in the left image coordinate system. The specific expression is:
[0140]
[0141] Further calculations yield:
[0142]
[0143] The value of the bottom y-coordinate of the feature point at the bottom of the first obstacle in the left image coordinate system is:
[0144] y=abs((zd*f*k1*k2 / Z-k1*b2+b1*k2) / (k2-k1))
[0145] The value of the bottom horizontal coordinate x1 of the bottom feature point of the first obstacle in the left image coordinate system is:
[0146]
[0147] Therefore, the bottom coordinates of the bottom feature points can be obtained.
[0148] Step 106: Convert the bottom coordinates into physical coordinates of the obstacle based on the sensor parameters and the depth value.
[0149] As an optional implementation manner, step 106 of converting the bottom coordinates into physical coordinates of the obstacle based on the sensor parameters and the depth value may specifically include:
[0150] Determine the center point coordinates of the left image;
[0151] Decentralize the bottom coordinates based on the center point coordinates to obtain target bottom coordinates;
[0152] The target bottom coordinates are converted into obstacle physical coordinates based on the depth value and the focal length.
[0153] In this embodiment, the target bottom coordinates of the center point coordinates of the left image can be converted into the physical coordinates of the obstacle according to the corresponding relationship between the depth value and the focal length, so that the physical coordinates of the obstacle are more consistent with the actual position.
[0154] In the embodiment of the present application, the center point coordinate c of the left image x The coordinates can be based on the left image coordinate system. The bottom coordinates are decentralized based on the center point coordinates. The calculation formula for the target bottom coordinates can be:
[0155] x1=x1-c x
[0156] Where x1 minus the principal point c x , convert the x1 coordinate from the left image coordinate system to the central image coordinate system with the image center as the origin; thereby achieving the decentralization of the bottom coordinate.
[0157] In the embodiment of the present application, the calculation formula for converting the target bottom coordinates into the physical coordinates of the obstacle can be:
[0158]
[0159] The x obtained at this time is the horizontal coordinate of the obstacle's physical coordinates, and the vertical coordinate of the target's bottom coordinate is the vertical coordinate of the obstacle's physical coordinates. The physical coordinates of the second obstacle can be obtained in the same way, which will not be repeated here.
[0160] By implementing the above steps 101 to 106, the detection frame of the obstacle can be identified more accurately without being affected by environmental changes, and the information of the detection frame can be processed to determine the position information of the obstacle relative to the terminal device, thereby improving the accuracy of the binocular vision positioning results. In addition, the present application can also improve the efficiency of obstacle recognition. In addition, the present application can also improve the accuracy of the disparity values contained in the disparity value set. In addition, the present application can also improve the accuracy of depth value calculation. In addition, the present application can also improve the accuracy of the coordinate calculation of the bottom feature point. In addition, the present application can also make the physical coordinates of the obstacle more closely match the actual orientation.
[0161] The present application also provides an application scenario, which applies the above-mentioned binocular vision positioning method based on deep learning. Specifically: the binocular vision positioning method based on deep learning provided in this embodiment can be applied in intelligent robot motion scenarios. During the movement of the intelligent robot, it is necessary to identify and locate obstacles on the road. The binocular vision positioning method based on deep learning provided in this embodiment belongs to the link of locating obstacles in the process of movement of the intelligent robot. Specifically, in the process of locating the obstacle, the detection frame of the obstacle can be identified more accurately, and then the information of the detection frame can be processed to determine the position information of the obstacle relative to the terminal device, thereby improving the accuracy of the binocular vision positioning result.
[0162] Based on the same inventive concept, the embodiments of the present application also provide a deep learning-based binocular vision positioning device for implementing the deep learning-based binocular vision positioning method involved above. The device is applied to a terminal device, on which a binocular vision sensor is provided, and the binocular vision sensor includes a left vision sensor and a right vision sensor. The implementation solution provided by the device to solve the problem is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more embodiments of the deep learning-based binocular vision positioning device provided below can be referred to the limitations of the deep learning-based binocular vision positioning method above, and will not be repeated here.
[0163] In an exemplary embodiment, Figure 4 As shown, a binocular vision positioning device based on deep learning is provided, including:
[0164] The acquisition unit 401 is configured to acquire left-side obstacle data and right-side obstacle data using the binocular vision sensor; wherein the left-side obstacle data includes a left-side image, which includes an image of the first obstacle and an image of the second obstacle; and the right-side obstacle data includes a right-side image, which includes an image of the first obstacle and an image of the second obstacle.
[0165] a first processing unit 402 configured to process the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle and second left detection frame information and second right detection frame information of the second obstacle; wherein the inference engine is built based on a deep learning model;
[0166] A second processing unit 403 is configured to obtain a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information;
[0167] A first calculation unit 404 is configured to calculate a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor;
[0168] A second calculation unit 405 is configured to calculate the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters;
[0169] The conversion unit 406 is configured to convert the bottom coordinates into physical coordinates of the obstacle based on the sensor parameters and the depth value.
[0170] As an optional implementation, a method for constructing an inference engine includes:
[0171] The constructed deep learning model is trained based on the acquired obstacle dataset to obtain a trained deep learning model;
[0172] Obtaining a weight file of the deep learning model; wherein the weight file includes parameters and weight data of the deep learning model;
[0173] Convert the weight file into a wst file;
[0174] The wst file is sent to the terminal device, and the wst file is converted into an engine file on the terminal device, so that the terminal device builds an inference engine based on the engine file.
[0175] Among them, by implementing this implementation method, the deep learning model can be trained according to the obtained obstacle data set to obtain a deep learning model with accurate recognition results; and the weight file in the deep learning model can be converted into a wst file that can be applied by the terminal device, so that the terminal device can build an inference engine based on the wst file, so that the terminal device can accurately identify the relevant information of the obstacle based on the inference engine, without sending the collected obstacle image to the service device for identification, thereby improving the efficiency of obstacle identification.
[0176] As an optional implementation manner, the second processing unit 403 obtains a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information by:
[0177] Performing edge detection on the first left detection frame information and the first right detection frame information to obtain a first contour line segment of the first obstacle and a second contour line segment of the second obstacle;
[0178] determining a first edge segment of the first obstacle from the first contour segment;
[0179] determining a second edge segment of the second obstacle from the second contour segment;
[0180] Performing edge detection on the second left detection frame information to obtain a third edge segment corresponding to the first edge segment;
[0181] Performing edge detection on the second right detection frame information to obtain a fourth edge segment corresponding to the second edge segment;
[0182] Acquire a plurality of first feature points from the first edge segment, and acquire a third feature point corresponding to the first feature point from the third edge segment;
[0183] Calculating a first disparity value between each of the first feature points and a third feature point corresponding to the first feature point;
[0184] Acquire multiple second feature points from the second edge segment, and acquire fourth feature points corresponding to the second feature points from the fourth edge segment;
[0185] calculating a second disparity value between each second feature point and a fourth feature point corresponding to the second feature point;
[0186] A plurality of first disparity values and a plurality of second disparity values are added to a set of disparity values between the left image and the right image.
[0187] Among them, when implementing this embodiment, edge detection can be first performed on the first left detection frame information and the first right detection frame information to determine the first edge segment of the first obstacle and the second edge segment of the second obstacle, and the third edge segment corresponding to the first edge segment and the fourth edge segment corresponding to the second edge segment can be determined from the second left detection frame information and the second right detection frame information; and then, a disparity value set can be calculated based on the position corresponding feature points obtained from the first edge segment and the third edge segment, and based on the position corresponding feature points obtained from the second edge segment and the fourth edge segment, thereby improving the accuracy of the disparity values contained in the disparity value set.
[0188] As an optional implementation manner, the first calculation unit 404 calculates the depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and the sensor parameters of the binocular vision sensor in the following manner:
[0189] Obtaining a baseline length, a focal length of the left vision sensor, a left view principal point physical coordinate of a left view principal point of the left vision sensor, and a right view principal point physical coordinate of a right view principal point of the right vision sensor from sensor parameters of the binocular vision sensor; wherein the baseline length is the distance between the left vision sensor and the right vision sensor;
[0190] Calculating each disparity value in the disparity value set based on the baseline length, the focal length, the physical coordinates of the left view principal point, and the physical coordinates of the right view principal point to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor; wherein the initial depth value set includes a plurality of initial depth values;
[0191] Determining the number of depth values of the initial depth value included in the initial depth value set;
[0192] Based on the number of depth values, an average of the initial depth values in the depth value set is calculated to obtain a depth value from the first obstacle to the optical center of the left vision sensor.
[0193] Among them, by implementing this embodiment, each disparity value in the disparity value set can be calculated based on the acquired sensor parameters of the binocular vision sensor, together with the physical coordinates of the principal point of the left view and the physical coordinates of the principal point of the right view, to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor, and then the average value of the initial depth value set can be determined as the depth value from the first obstacle to the optical center of the left vision sensor, thereby improving the accuracy of the depth value calculation.
[0194] As an optional implementation manner, the second calculation unit 405 calculates the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters in the following manner:
[0195] Obtain a first line segment parameter of a first expression of the first edge line segment;
[0196] Obtain a second line segment parameter of a second expression of the second edge line segment;
[0197] collecting the current distance between the first obstacle and the second obstacle by a preset distance measuring sensor;
[0198] Calculate the bottom vertical coordinate of the bottom feature point of the first obstacle in the left image based on the first line segment parameter, the second line segment parameter, the current distance, the depth value, and the focal length;
[0199] Calculate the bottom horizontal coordinate of the bottom feature point based on the bottom vertical coordinate and the first expression;
[0200] The bottom horizontal coordinate and the bottom vertical coordinate are combined to obtain the bottom coordinate of the bottom feature point.
[0201] Among them, by implementing this embodiment, calculations can be performed based on the parameters of the first expression of the first edge segment, the parameters of the second expression of the second edge segment, the current distance, depth value and focal length collected between the first obstacle and the second obstacle, and the horizontal and vertical coordinates of the bottom feature point of the first obstacle in the left image can be obtained, thereby improving the accuracy of the coordinate calculation of the bottom feature point.
[0202] As an optional implementation manner, the conversion unit 406 converts the bottom coordinates into the physical coordinates of the obstacle based on the sensor parameters and the depth value in the following manner:
[0203] Determine the center point coordinates of the left image;
[0204] Decentralize the bottom coordinates based on the center point coordinates to obtain target bottom coordinates;
[0205] The target bottom coordinates are converted into obstacle physical coordinates based on the depth value and the focal length.
[0206] In this embodiment, the target bottom coordinates of the center point coordinates of the left image can be converted into the physical coordinates of the obstacle according to the corresponding relationship between the depth value and the focal length, so that the physical coordinates of the obstacle are more consistent with the actual position.
[0207] By implementing this embodiment, it is possible to more accurately identify the detection frame of the obstacle without being affected by environmental changes, and then process the information of the detection frame to determine the position information of the obstacle relative to the terminal device, thereby improving the accuracy of the binocular vision positioning results. In addition, the present application can also improve the efficiency of obstacle recognition. In addition, the present application can also improve the accuracy of the disparity values contained in the disparity value set. In addition, the present application can also improve the accuracy of the depth value calculation. In addition, the present application can also improve the accuracy of the coordinate calculation of the bottom feature point. In addition, the present application can also make the physical coordinates of the obstacle more closely match the actual orientation.
[0208] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a binocular visual positioning method based on deep learning is implemented.
[0209] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0210] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.
[0211] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0212] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0213] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0214] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0215] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.
[0216] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0217] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A binocular vision positioning method based on deep learning, characterized in that: The method is applied to a terminal device, wherein a binocular vision sensor is provided on the terminal device, wherein the binocular vision sensor includes a left vision sensor and a right vision sensor. The binocular vision positioning method based on deep learning includes: Collecting left-side obstacle data and right-side obstacle data using the binocular vision sensor; wherein the left-side obstacle data includes a left-side image, which includes an image of the first obstacle and an image of the second obstacle; and the right-side obstacle data includes a right-side image, which includes an image of the first obstacle and an image of the second obstacle; Processing the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle and second left detection frame information and second right detection frame information of the second obstacle; wherein the inference engine is built based on a deep learning model; Obtaining a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information; Calculating a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor; Calculate the bottom coordinates of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters; The bottom coordinates are converted into physical coordinates of an obstacle based on the sensor parameters and the depth value.
2. The binocular vision positioning method based on deep learning according to claim 1, characterized in that The method for constructing the inference engine includes: The constructed deep learning model is trained based on the acquired obstacle dataset to obtain a trained deep learning model; Obtaining a weight file of the deep learning model; wherein the weight file includes parameters and weight data of the deep learning model; Convert the weight file into a wst file; The wst file is sent to the terminal device, and the wst file is converted into an engine file on the terminal device, so that the terminal device builds an inference engine based on the engine file.
3. The binocular vision positioning method based on deep learning according to claim 1 or 2, characterized in that: The obtaining, based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information, a set of disparity values between the left image and the right image specifically includes: Performing edge detection on the first left detection frame information and the first right detection frame information to obtain a first contour line segment of the first obstacle and a second contour line segment of the second obstacle; determining a first edge segment of the first obstacle from the first contour segment; determining a second edge segment of the second obstacle from the second contour segment; Performing edge detection on the second left detection frame information to obtain a third edge segment corresponding to the first edge segment; Performing edge detection on the second right detection frame information to obtain a fourth edge segment corresponding to the second edge segment; Acquire a plurality of first feature points from the first edge segment, and acquire a third feature point corresponding to the first feature point from the third edge segment; Calculating a first disparity value between each of the first feature points and a third feature point corresponding to the first feature point; Acquire multiple second feature points from the second edge segment, and acquire fourth feature points corresponding to the second feature points from the fourth edge segment; calculating a second disparity value between each second feature point and a fourth feature point corresponding to the second feature point; A plurality of first disparity values and a plurality of second disparity values are added to a set of disparity values between the left image and the right image.
4. The binocular vision positioning method based on deep learning according to claim 3, characterized in that: The calculating, based on the disparity value set and the sensor parameters of the binocular vision sensor, a depth value from the first obstacle to the optical center of the left vision sensor specifically includes: Obtaining a baseline length, a focal length of the left vision sensor, a left view principal point physical coordinate of a left view principal point of the left vision sensor, and a right view principal point physical coordinate of a right view principal point of the right vision sensor from sensor parameters of the binocular vision sensor; wherein the baseline length is the distance between the left vision sensor and the right vision sensor; Calculating each disparity value in the disparity value set based on the baseline length, the focal length, the physical coordinates of the left view principal point, and the physical coordinates of the right view principal point to obtain an initial depth value set from the first obstacle to the optical center of the left vision sensor; wherein the initial depth value set includes a plurality of initial depth values; Determining the number of depth values of the initial depth value included in the initial depth value set; Based on the number of depth values, an average of the initial depth values in the depth value set is calculated to obtain a depth value from the first obstacle to the optical center of the left vision sensor.
5. The binocular vision positioning method based on deep learning according to claim 4, characterized in that: The calculating, based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameter, a bottom coordinate of the first obstacle in the left image specifically includes: Obtain a first line segment parameter of a first expression of the first edge line segment; Obtain a second line segment parameter of a second expression of the second edge line segment; collecting the current distance between the first obstacle and the second obstacle by a preset distance measuring sensor; Calculate the bottom vertical coordinate of the bottom feature point of the first obstacle in the left image based on the first line segment parameter, the second line segment parameter, the current distance, the depth value, and the focal length; Calculate the bottom horizontal coordinate of the bottom feature point based on the bottom vertical coordinate and the first expression; The bottom horizontal coordinate and the bottom vertical coordinate are combined to obtain the bottom coordinate of the bottom feature point.
6. The binocular vision positioning method based on deep learning according to claim 4, characterized in that: The converting the bottom coordinates into physical coordinates of an obstacle based on the sensor parameters and the depth value specifically includes: Determine the center point coordinates of the left image; Decentralize the bottom coordinates based on the center point coordinates to obtain target bottom coordinates; The target bottom coordinates are converted into obstacle physical coordinates based on the depth value and the focal length.
7. A binocular vision positioning device based on deep learning, characterized in that: The device is applied to a terminal device, wherein a binocular vision sensor is provided on the terminal device, wherein the binocular vision sensor includes a left vision sensor and a right vision sensor. The binocular vision positioning device based on deep learning includes: a collection unit configured to collect left-side obstacle data and right-side obstacle data using the binocular vision sensor; wherein the left-side obstacle data includes a left-side image, which includes an image of the first obstacle and an image of the second obstacle; and the right-side obstacle data includes a right-side image, which includes an image of the first obstacle and an image of the second obstacle. a first processing unit, configured to process the left obstacle data and the right obstacle data based on a pre-built inference engine to obtain first left detection frame information and first right detection frame information of the first obstacle, and second left detection frame information and second right detection frame information of the second obstacle; wherein the inference engine is built based on a deep learning model; a second processing unit, configured to obtain a set of disparity values between the left image and the right image based on the first left detection frame information, the first right detection frame information, the second left detection frame information, and the second right detection frame information; a first calculating unit, configured to calculate a depth value from the first obstacle to the optical center of the left vision sensor based on the disparity value set and sensor parameters of the binocular vision sensor; a second calculation unit, configured to calculate a bottom coordinate of the first obstacle in the left image based on the first left detection frame information, the first right detection frame information, the depth value, and the sensor parameters; A conversion unit is configured to convert the bottom coordinates into physical coordinates of an obstacle based on the sensor parameters and the depth value.
8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the binocular vision positioning method based on deep learning according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the binocular vision positioning method based on deep learning described in any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the binocular vision positioning method based on deep learning described in any one of claims 1 to 6 are implemented.