Monocular distance measurement method, apparatus and device, and computer readable storage medium

Abstract

The invention discloses a monocular distance measurement method, device and equipment and a computer readable storage medium, and the method comprises the steps: carrying out the detection of a target object in a target image collected by image collection equipment through a deep learning algorithm, and determining a distance relation and an orientation relation between the target object and the image collection equipment according to a detection result, and constructing an observation equation of the target object relative to the image acquisition equipment according to the distance relationship and the orientation relationship, determining a target value of a state variable of the target object based on the observation equation and a pre-constructed extended Kalman filtering distance measurement model, and determining the distance between the target object and the image acquisition equipment according to the target value of the state variable. According to the embodiment, the problem that existing monocular distance measurement is not accurate enough can be solved.

Description

Technical Field

[0001] The present application belongs to the field of distance measurement, and in particular, relates to a monocular distance measurement method, device, equipment and computer-readable storage medium. Background Art

[0002] Monocular ranging refers to solving the position of an object in the physical coordinate system based on the geometric relationship of the camera imaging, and then obtaining the distance of the object. At present, monocular ranging is widely used, for example, in advanced assisted driving such as autonomous driving systems and ADAS (Advanced Driving Assistance System). The current monocular ranging method mainly uses 2D frames for ranging, including: 1. plane assumption, 2. prior size and other methods. Among them, the plane assumption is to assume that the ground plane is flat, and the bottom of the 2D frame falls on the plane. The plane assumption has good accuracy at close range, but the long-distance ranging has large errors due to the shaking of the vehicle body and the undulations of the ground. Although the prior size generally has good stability, it is very dependent on the size of the assumed size. If the size is set unreasonably, it will cause ranging deviations and low accuracy. Summary of the invention

[0003] The embodiments of the present application provide a monocular ranging method, device, equipment and computer-readable storage medium, which can improve the problem of low accuracy of existing monocular ranging methods.

[0004] In a first aspect, an embodiment of the present application provides a monocular ranging method, the method comprising:

[0005] Detecting the target object in the target image captured by the image acquisition device through a deep learning algorithm to obtain a detection result;

[0006] Determine the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection results;

[0007] An observation equation of the target object relative to the image acquisition device is constructed according to the distance relationship and the orientation relationship, wherein the observation equation corresponds to a state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device;

[0008] Determine the target value of the state variable based on the observation equation and the pre-constructed extended Kalman filter ranging model, the extended Kalman filter ranging model corresponds to the state variable;

[0009] The distance between the target object and the image acquisition device is determined according to the target value of the state variable.

[0010] In a second aspect, an embodiment of the present application provides a monocular ranging device, the device comprising:

[0011] The target detection module is used to detect the target object in the target image acquired by the image acquisition device through a deep learning algorithm to obtain a detection result;

[0012] A relationship determination module, used to determine the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result;

[0013] A first construction module is used to construct an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship, wherein the observation equation corresponds to a state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device;

[0014] A target value determination module is used to determine the target value of the state variable based on the observation equation and the pre-built extended Kalman filter ranging model, the extended Kalman filter ranging model corresponding to the state variable;

[0015] The distance determination module is used to determine the distance between the target object and the image acquisition device according to the target value of the state variable.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0017] When the processor executes the computer program instructions, the monocular ranging method of the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the monocular ranging method of the first aspect is implemented.

[0019] The monocular ranging method, apparatus, device and computer-readable storage medium of the embodiment of the present application detects the target object in the target image captured by the image acquisition device through a deep learning algorithm to obtain a detection result, determines the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result, constructs an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship, determines the target value of the state variable based on the observation equation and a pre-constructed extended Kalman filter ranging model, and determines the distance between the target object and the image acquisition device according to the target value of the state variable. According to this embodiment, compared with the traditional ranging method based on the plane assumption, different observation equations are constructed for different distance relationships and orientation relationships, so as to achieve accurate ranging in different scenes. Compared with the ranging method based on a priori size, this embodiment uses the extended Kalman filter ranging model for ranging, which is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0022] Figure 2 It is a flow chart of the monocular ranging method provided in the embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a target object detection result in a target image provided by an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a target object detection result in another target image provided by an embodiment of the present application;

[0025] FIG5( a ) is a schematic diagram of a target object detection result in a target image provided by an embodiment of the present application;

[0026] FIG5( b ) is a schematic diagram of a hypothetical motion trajectory of a target object provided in an embodiment of the present application;

[0027] Figure 6 is a structural schematic diagram of a monocular distance measuring device provided in an embodiment of the present application;

[0028] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0031] The embodiments of the present application provide a monocular ranging method, device, equipment, medium and program.

[0032] First, a monocular ranging method provided in an embodiment of the present application is described.

[0033] See also Figure 1 , is an exemplary system architecture to which the monocular ranging method provided by the embodiment of the present application can be applied, such as Figure 1 As shown, the system architecture may include an image acquisition device 110, a server 120, and a network 130, wherein the network 130 is used to provide a medium for a communication link between the image acquisition device 110 and the server 120. The image acquisition device 110 exchanges data with the server 120 via the network 130. The network 130 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] The image acquisition device 110 is an electronic device with an image acquisition function, including but not limited to various cameras, etc.

[0035] The server 120 may be a server that provides various services, including but not limited to a vehicle-mounted computer, a cloud server, etc. The server 120 may analyze and process the images captured by the image acquisition device 110 .

[0036] The monocular distance measurement method provided in the embodiment of the present application may be executed by the server 120. Accordingly, the monocular distance measurement device may be disposed in the server 120.

[0037] It should be understood that Figure 1 The number of image acquisition devices and servers in the embodiment is only for illustration. Any number of image acquisition devices and servers may be provided according to implementation requirements.

[0038] The monocular ranging method provided in the embodiment of the present application can be applied to automatic driving systems, ADAS and other systems in assisted driving scenarios. When applied to assisted driving scenarios, the image acquisition device 110 can be installed on the body of the vehicle to collect images around the vehicle, and the server 120 can use the on-board computer in the vehicle.

[0039] See also Figure 2 , is a flow chart of the monocular ranging method provided in the embodiment of the present application, such as Figure 2 As shown, the method may include the following steps S21-S25.

[0040] S21. Detect the target object in the target image captured by the image acquisition device through a deep learning algorithm to obtain a detection result.

[0041] The monocular distance measurement method provided in the embodiment of the present application is used to measure the distance between a target object and an image acquisition device.

[0042] The image acquisition device may include a camera or other device with an image acquisition function.

[0043] The target image refers to the image containing the target object captured by the image acquisition device.

[0044] The target object can be determined according to the actual application scenario. For example, in an assisted driving scenario, the target object can be a vehicle.

[0045] After acquiring the target image captured by the image acquisition device, image detection is performed on the target object in the target image based on the deep learning algorithm to obtain a detection result.

[0046] S22. Determine the distance relationship and orientation relationship between the target object and the image acquisition device based on the detection results.

[0047] The distance relationship is used to indicate whether the target object is near or far from the image acquisition device, and the orientation relationship is used to indicate whether the target object is facing the image acquisition device.

[0048] S23. Construct an observation equation of the target object relative to the image acquisition device based on the distance relationship and the orientation relationship. The observation equation corresponds to the state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device.

[0049] Because the detection results of the target object have different influences on the value of the state variable under different distance and orientation relationships, in order to improve the accuracy of the ranging results, different observation equations are constructed according to different distance and orientation relationships.

[0050] Before constructing the observation equation, the state variable corresponding to the target object is pre-constructed, thereby constructing the observation equation corresponding to the state variable, so that the value of the state variable can be determined according to the observation equation. Among them, because the ultimate goal of this embodiment is to obtain the distance between the target object and the image acquisition device, the state variable needs to be related to the distance between the target object and the image acquisition device.

[0051] S24. Determine the target value of the state variable based on the observation equation and the pre-constructed extended Kalman filter ranging model, and the extended Kalman filter ranging model corresponds to the state variable.

[0052] Before performing ranging, an extended Kalman filter ranging model is constructed based on the state variables of the target object. The extended Kalman filter ranging model is a model constructed based on the extended Kalman filter algorithm for determining the state variable value of the target object.

[0053] The extended Kalman filter algorithm includes two steps, namely a prediction step and an update step, wherein the prediction step is used to predict the value of the state variable, and the update step is used to correct and update the predicted value of the state variable obtained in the prediction step. Based on this, the extended Kalman filter ranging model can include a prediction model for implementing the prediction step and an update model for implementing the update step. Among them, the prediction model is used to predict the value of the state variable based on the motion equation of the target object relative to the image acquisition device, and the update model is used to correct the prediction result output by the prediction model based on the observation equation of the target object relative to the image acquisition device, so as to obtain the corrected accurate state variable value, and the corrected state variable value is used as the target value of the state variable. Because the motion equation of the target object relative to the image acquisition device is not affected by the target object detection result, it can be pre-set and directly called during ranging.

[0054] S25. Determine the distance between the target object and the image acquisition device according to the target value of the state variable.

[0055] Because the state variable is related to the distance between the target object and the image acquisition device, and the correlation is known, after obtaining the target value of the state variable, the distance between the target object and the image acquisition device can be determined based on the known correlation, thereby completing monocular ranging.

[0056] The present embodiment provides a monocular ranging method, which detects a target object in a target image captured by an image acquisition device through a deep learning algorithm to obtain a detection result, determines the distance relationship and orientation relationship between the target object and the image acquisition device based on the detection result, constructs an observation equation of the target object relative to the image acquisition device based on the distance relationship and orientation relationship, obtains a target value of a state variable based on the observation equation and a pre-constructed extended Kalman filter ranging model, and determines the distance between the target object and the image acquisition device based on the target value of the state variable. According to the present embodiment, compared with the traditional ranging method based on the plane assumption, different observation equations are constructed for different distance relationships and orientation relationships, thereby achieving accurate ranging in different scenarios. Compared with the ranging method based on a priori size, the present embodiment uses the extended Kalman filter ranging model for ranging, which is more accurate.

[0057] In some embodiments, the detection result may include a two-dimensional target frame, bottom edge, and side edge corresponding to the target object. The two-dimensional target frame is a target frame used to indicate that the selected target is the target object, the bottom edge of the target object refers to the straight line where the bottom of the target object contacts the ground, and the side edge refers to a straight line perpendicular to the ground and passing through the corners of the target object.

[0058] As an example, taking the target object as a vehicle, its bottom edge refers to the straight line where the intersection of the tires on the same side and the ground is located, such as the straight line where the intersection of the left front wheel and the ground and the intersection of the left rear wheel and the ground are located, the straight line where the intersection of the right front wheel and the ground and the intersection of the right rear wheel and the ground are located, the straight line where the intersection of the left front wheel and the ground and the intersection of the right front wheel and the ground are located, and the straight line where the intersection of the left rear wheel and the ground and the intersection of the right rear wheel and the ground are located. Its side edges refer to the straight line that is perpendicular to the ground and passes through the leftmost end of the front of the vehicle, the straight line that is perpendicular to the ground and passes through the rightmost end of the front of the vehicle, the straight line that is perpendicular to the ground and passes through the leftmost end of the rear of the vehicle, and the straight line that is perpendicular to the ground and passes through the rightmost end of the rear of the vehicle. See Figure 3 , which is a schematic diagram of the detection result of a target object in a target image, in which two side edges coincide with the two-dimensional target box.

[0059] Currently, there are many deep learning algorithms for image target detection. This embodiment can use any deep learning algorithm that can detect two-dimensional target boxes and the edges of target objects to detect target objects in the target image, including but not limited to R-CNN image target detection model, edge detection algorithm, etc.

[0060] In the case where the detection result includes a two-dimensional target frame, a bottom edge, and a side edge corresponding to the target object, determining the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result may include:

[0061] Determine the distance relationship between the target object and the image acquisition device according to the two-dimensional target frame;

[0062] The orientation relationship between the target object and the image acquisition device is determined according to the number of side edges.

[0063] In the embodiment of the present application, the distance relationship between the target object and the image acquisition device may include the target object being near the image acquisition device and the target object being far from the image acquisition device. Exemplarily, when determining the distance relationship between the target object and the image acquisition device according to the two-dimensional target frame, the following steps may be included:

[0064] Determine whether the target object in the two-dimensional target box is truncated;

[0065] In the case where it is determined that the target object in the two-dimensional target frame is truncated, it is determined that the target object is located near the image acquisition device;

[0066] When it is determined that the target object in the two-dimensional target frame is not truncated, it is determined that the target object is located far away from the image acquisition device.

[0067] The target object being truncated means that the target object is located at the edge of the target image and is truncated by the target image. In practical applications, whether the target object is truncated can be determined according to the truncation property of the two-dimensional target frame.

[0068] In the embodiment of the present application, the orientation relationship between the target object and the image acquisition device may include the target object facing the image acquisition device and the target object not facing the image acquisition device. Exemplarily, when determining the orientation relationship between the target object and the image acquisition device according to the number of side edges, the following steps may be included:

[0069] Determine whether the number of side edges is less than three;

[0070] When the number of side edges is less than three, it is determined that the target object is facing the image acquisition;

[0071] When the number of the side edges is greater than or equal to three, it is determined that the target object is not facing the image acquisition device.

[0072] The target object is facing the image acquisition device, which means that the target object is on a circle with the optical center of the image acquisition device as the center.

[0073] As an example, take the target object as a vehicle, see Figure 4 In the target image shown, only two side edges can be detected, and these two side edges coincide with the sides of the two-dimensional target frame. Therefore, it is considered that the target object is located facing the image acquisition device.

[0074] Through the above method, the distance relationship and the orientation relationship between the target object and the image acquisition device can be accurately determined.

[0075] The principle of the monocular ranging method provided in the embodiment of the present application is to determine the target value of the state variable of the target object based on the extended Kalman filter algorithm (Extended Kalman Filter), and then determine the distance between the target object and the image acquisition device based on the target value of the state variable. When determining the target value of the state variable based on the extended Kalman filter algorithm, it is necessary to use the extended Kalman filter ranging model, the observation equation of the target object relative to the image acquisition device, and the motion equation of the target object relative to the image acquisition device. Because the construction of the motion equation and the extended Kalman filter ranging model are not affected by the detection result of the target object, the motion equation and the extended Kalman filter ranging model can be pre-constructed, and can be directly obtained when performing monocular ranging.

[0076] In the embodiment of the present application, the extended Kalman filter distance measurement model, the observation equation of the target object relative to the image acquisition device, and the motion equation of the target object relative to the image acquisition device are all state variables of the target object, so before constructing the model and the equation, the state variables of the target object can be constructed first. When constructing the state variables of the target object, the parameters that affect the distance between the target object and the image acquisition device can be determined according to the characteristics of the target object, and the state variables can be selected from them.

[0077] In some embodiments, the state variables of the target object may include: position coordinates, speed, orientation angle, rotation angular velocity, width, and length. t =[P t , v t ,θ t , w t , W t , L t ] indicates that, among them, P t Indicates the position coordinates of the target object, including the forward x coordinate, the horizontal y coordinate, and the vertical z=0, v t Represents the velocity scalar of the target object, θ t Represents the target object's orientation angle scalar, only on the xoy plane, w t Represents the angular velocity of the target object, W t Indicates the width of the target object, L t Indicates the length of the target object. The xoy plane refers to a plane in three-dimensional space. The xoy plane is an infinitely extended plane, which consists of all points that satisfy z = 0. In a three-dimensional coordinate system, the x-axis and the y-axis coincide with the xoy plane, while the z-axis is perpendicular to the xoy plane.

[0078] Since the target object usually refers to a class of objects, such as vehicles, people, animals, etc., and the scales of the objects in the same class may be inconsistent, if the size is set by experience, it may deviate from the actual size, resulting in the final distance measurement result being inaccurate. Therefore, in the embodiment of the present application, W t and L t Putting it into the state variable for optimization can make the final ranging result more accurate.

[0079] When constructing an extended Kalman filter ranging model, an extended Kalman filter ranging model corresponding to the state variable can be constructed based on an extended Kalman filter algorithm.

[0080] The extended Kalman filter ranging model includes a prediction model and an update model. The prediction model is used to predict state variables based on the motion equation of the target object relative to the image acquisition device, and the update model is used to correct the results output by the prediction model based on the observation equation of the target object relative to the image acquisition device.

[0081] Among them, the prediction model may include:

[0082]

[0083]

[0084] Where f represents the motion equation of the target object relative to the image acquisition device, F represents the Jacobian matrix of the motion equation, P represents the covariance matrix of the state edge wave, Q represents the prediction noise, and u k represents the control variable. In this embodiment, u k The default is 0.

[0085] Updating a model can include:

[0086]

[0087]

[0088]

[0089]

[0090] P k+1∣k+1 =(IK k+1 H k+1 ) k+1∣k

[0091] Where z represents the observation value, h represents the observation equation of the target object relative to the image acquisition device, H represents the Jacobian matrix corresponding to the observation equation, and K represents the extended Kalman gain.

[0092] When constructing the motion equation, the motion equation of the target object relative to the image acquisition device can be constructed based on the assumption that the target object moves in an assumed xoy plane and satisfies the constant speed and constant angular velocity.

[0093] Based on the above state variables, its discrete kinematic model, that is, the motion equation, satisfies the following formula:

[0094]

[0095] v(t+Δt)=v(t)

[0096] θ(t+Δt)=θ(t)+w(t)Δt

[0097] w(t+Δt)=w(t)

[0098] W(t+Δt)=W(t)

[0099] L(t+Δt)=L(t)

[0100] In terms of observation equations, the embodiments of the present application can construct observation equations based on the distance relationship and orientation relationship between the target object and the image acquisition device using different observation methods.

[0101] In some embodiments, the observation method can be determined based on the following three different assumption modes:

[0102] Assumption 1: When the target object is close to the image acquisition device, it is assumed that the ground plane has no undulations and the target object jitter is very small. The errors of these two assumptions are very small at close range, so the conditions are met.

[0103] Assumption 2: When the target object is far away from the image acquisition device and is not facing the image acquisition device, it is assumed that the target object moves on a plane, that is, the longitudinal coordinate is 0, and only the yaw angle changes.

[0104] Assumption 3: When the target object is far away from the image acquisition device and is facing the image acquisition device, it is assumed that the target object is on a circle with the optical center of the image acquisition device as the center. Three different assumptions correspond to three different observation equations.

[0105] Based on the above assumption 1, when the target object is close to the image acquisition device, the observation equation can be established based on the IPM (Inverse Perspective Mapping) observation method. At present, the jitter of the target object and the error of the plane assumption are very small in the close distance situation, which basically meets the assumption condition (1. In this case, there are mainly P tThe forward x coordinate, the lateral y coordinate and θ are three variables, so three observation equations need to be established. The three observation equations include: the first observation equation corresponding to the horizontal coordinate of the target side edge at the projection point on the ground, the second observation equation corresponding to the vertical coordinate of the target side edge at the projection point on the ground, and the third observation equation corresponding to the slope of the bottom edge. The target side edge is the side edge detected in the target image. The first observation equation, the second observation equation and the third observation equation are used as observation equations of the target object relative to the image acquisition device.

[0106] First, define the rotation and translation matrix from the ground coordinate system to the camera coordinate system of the image acquisition device as T = [R|t], and the rotation and translation matrix from the camera coordinate system of the image acquisition device to the ground coordinate system as T c2w =[R T |-R T t]=[R T |C], In order to facilitate subsequent calculations, this embodiment translates the ground coordinate system to the position of the image acquisition device. In the new coordinate system, the internal and external parameters of the image acquisition device are combined to obtain Among them, A1 represents the internal and external parameters in the x-axis direction, A2 represents the internal and external parameters in the y-axis direction, and A3 represents the internal and external parameters in the z-axis direction. Therefore, AP w =P uv , where P w Represents the world coordinate point, P uv Represents the pixel coordinates. This formula represents the conversion of the world coordinate system point coordinates to the pixel coordinate system.

[0107] Using the mapping from world coordinates to pixel coordinates, according to the affine transformation, the straight line in the world coordinate system is still a straight line when projected into the image, and at the same time satisfies a mapping relationship, the following observation equation can be established:

[0108]

[0109] In the formula, represents the coordinates of the projection point of the target side edge of the target object on the ground, x and y represent the distance coordinates from the optical center of the image acquisition device to the center of the target object, C represents the translation vector as a constant, Θ represents the coordinates of the edge projected on the ground with the center of the target object as the origin, u and v represent the pixel coordinates of the landing point of the target side edge in the target image, the landing point refers to the intersection of the target side edge and the bottom edge, k represents the slope of the bottom edge of the target object, where k corresponds to the variable B1=(A3P k A2-A2P k A3)P1,B2=(A3P k A1-A2P k A1)P1, where P1 represents the projection point corresponding to the slope line,

[0110] The target object includes four side edges, namely the left front edge, the right front edge, the left rear edge and the right rear edge, where the coordinates of the projection point of each side edge on the ground are:

[0111]

[0112]

[0113]

[0114]

[0115] Where P lfd represents the coordinates of the projection point corresponding to the left front edge, P rfd represents the coordinates of the projection point corresponding to the right front edge, P lrd represents the coordinates of the projection point corresponding to the left rear edge, P rrd Indicates the projection point coordinates corresponding to the right rear edge. According to the target edge, the corresponding projection point coordinates are selected and substituted into the observation equation to obtain the first observation equation corresponding to the horizontal coordinate of the projection point of the target side edge on the ground, the second observation equation corresponding to the vertical coordinate of the projection point of the target side edge on the ground, and the third observation equation corresponding to the slope of the bottom edge, that is, the observation equation of the target object relative to the image acquisition device.

[0116] Based on the above assumption 2, when the target object is far away from the image acquisition device and is not facing the image acquisition device, the error of the IPM space distance measurement method is very large due to the jitter of the distant target object itself and the undulations of the road surface, and the assumption is no longer applicable. At this time, the observation equations corresponding to the horizontal coordinates of the three side edges in the target image are constructed to obtain three observation equations, which are used as the observation equations of the target object relative to the image acquisition device.

[0117] Taking the three side edges as the left front edge, the left rear edge, and the right rear edge as examples, three observation equations are established:

[0118]

[0119] In the formula, u lfd Indicates the horizontal coordinate of the point where the left front edge lands in the target image, u lrd Indicates the horizontal coordinate of the point where the left rear edge lands in the target image, u rrd Represents the horizontal coordinate of the point where the right rear edge lands in the target image.

[0120] The following observation equation corresponding to the left front edge is taken as an example, and its observation equation is as follows:

[0121]

[0122] Based on the above assumption 3, when the target object is far away from the image acquisition device and facing the image acquisition device, it is difficult to observe the three side edges. If the observation equation is constructed in the above manner, there will be one less observation equation corresponding to the side edge, and there will be more parameters to be solved than the equation group, resulting in no solution for the parameters. In this case, in order to ensure that the state parameters can be solved, it is assumed that the motion trajectory of the target object relative to the image acquisition device satisfies circular motion, that is, the two endpoints of the bottom edge of the target object are always on a circle with the optical center of the image acquisition device as the center, and the radius of the circle is the distance from the optical center to the corner of the target object.

[0123] Exemplarily, taking a vehicle as the target object, as shown in FIG5(a), the target image includes three target objects, and only two side edges can be detected for each of the three target objects, namely, the two sides of the two-dimensional target box in the figure. In this case, it is assumed that the motion trajectories of the three target objects relative to the image acquisition device all satisfy circular motion, as shown in FIG5(b), wherein the thin-line frame represents the actual target object, and the thick-line frame represents the hypothetical target object that moves in a circular motion. As shown in FIG5, it is assumed that the two endpoints of the bottom edge of the target object are both on a circle with the optical center of the image acquisition device as the center, and the radius of the circle is the distance from the optical center to the corner of the target object, and the straight line where the optical center and the center of the target object are located is orthogonal to the bottom edge.

[0124] Based on the above assumptions, observation equations corresponding to the horizontal coordinates of the two side edges in the target image are constructed to obtain two observation equations, and a constraint equation is constructed. The constraint equation is used to indicate that the bottom edge corresponding to the orientation of the target vehicle and the optical center of the image acquisition device satisfy the orthogonality. The two observation equations and the constraint equation are used as the observation equations of the target object relative to the image acquisition device.

[0125] Taking the two side edges as the left front edge and the right front edge as an example, the two observation equations and one constraint equation are as follows:

[0126]

[0127] Where θ1 and θ2 represent the projection coordinates of the target object's orientation corresponding to the bottom edge, D represents the distance coordinate from the optical center of the image acquisition device to the center of the target object, and θ1-θ2 satisfies orthogonality with D.

[0128] Simplifying the constraint equations, we get the following observation equations:

[0129]

[0130] In some embodiments, when determining the target value of the state variable based on the observation equation and the pre-built extended Kalman filter ranging model, the following steps may be included:

[0131] Determine the first Jacobian matrix of the observation equation;

[0132] Obtaining a second Jacobian matrix of a motion equation of the target object relative to the image acquisition device, the motion equation corresponding to the state variable;

[0133] The first Jacobian matrix and the second Jacobian matrix are input into the extended Kalman filter ranging model, and the prediction model in the extended Kalman filter ranging model is used to predict the state variable according to the first Jacobian matrix to obtain the predicted value of the state variable, and the update model in the extended Kalman filter ranging model is used to update the predicted value according to the second Jacobian matrix to obtain the target value of the state variable.

[0134] Among them, the Jacobian matrix of the observation equation, that is, the first Jacobian matrix, can be determined by taking partial derivatives of the observation equation.

[0135] Exemplarily, when the target object is located near the image acquisition device, the partial derivatives of u, v, and k in the observation equation can be solved respectively according to the following formulas:

[0136]

[0137]

[0138]

[0139] In this way, the first Jacobian matrix can be obtained as shown below:

[0140]

[0141] Take the target side edge as the left front edge as an example, where

[0142] When the target object is far away from the image acquisition device and not facing the image acquisition device, the partial derivatives of each state variable of the observation equation corresponding to each of the three side edges can be solved respectively, and then the first Jacobian matrix can be constructed according to the partial derivatives corresponding to the three side edges.

[0143] Taking the three side edges, namely the left front edge, the left rear edge, and the right rear edge, as an example, solving the partial derivative of each state variable in the observation equation corresponding to the left front edge, we can get:

[0144]

[0145] The method of solving the partial derivatives of each state variable in the observation equation corresponding to the left rear edge and the right rear edge is similar to the above method and will not be repeated here.

[0146] Finally, the first Jacobian matrix can be obtained as shown below:

[0147]

[0148]

[0149] When the target object is far away from the image acquisition device and facing the image acquisition device, the variables of the observation equations corresponding to the two side edges can be replaced according to the constraint equation, and then the partial derivatives of each state variable in each observation equation after the variable replacement are solved respectively, and then the first Jacobian matrix is ​​constructed according to the partial derivatives corresponding to the three side edges.

[0150] Taking two side edges including the left front edge and the right front edge as an example, solving the partial derivatives of each state variable in the observation equation corresponding to the two side edges respectively, we can get:

[0151]

[0152]

[0153] Similarly,

[0154]

[0155]

[0156] Based on the above, we can get the first Jacobian matrix as shown below:

[0157]

[0158] Similarly, the Jacobian matrix corresponding to the motion equation, that is, the second Jacobian matrix, can be obtained by solving the partial derivatives of the motion equation. The second Jacobian matrix can be shown as follows:

[0159]

[0160] After obtaining the first Jacobian matrix and the second Jacobian matrix, the first Jacobian matrix and the second Jacobian matrix are input into the extended Kalman filter ranging model to obtain the target value of the state variable output by the extended Kalman filter ranging model.

[0161] In summary, the embodiment of the present application provides a monocular ranging method based on tight coupling. First, the target object in the target image collected by the image acquisition device is detected by deep learning, and the detection results including the two-dimensional target frame and the bottom edge and side edge of the target object are obtained. Since the image naturally lacks scale information and lacks one dimension information in three dimensions, the result of deep learning detection is also difficult to restore the 3D properties of the target. There is some information loss in the monocular. In order to meet the range finding, this embodiment combines the detection results of deep learning and the timing information of the range finding for comprehensive range finding. The state variables such as the position coordinates, size, orientation angle and angular velocity of the target object are constructed, and the detection results of deep learning are used as observations to construct an extended Kalman filter ranging model for comprehensive range finding and filtering. According to the distance relationship and orientation relationship between the target object and the image acquisition device, a variety of ranging observation equations are constructed. For the target object in the near distance, the observation equation is established using the plane assumption of IPM, and for the target object in the distance, the observation equation is established using the size of the target object. At the same time, the size of the target object is put into the state variable, and the time series is used for regression. In this way, an accurate ranging result can be obtained in the end.

[0162] Based on the monocular distance measurement method provided in the above embodiment, the present application also provides a specific implementation of a monocular distance measurement device. Please refer to the following embodiment.

[0163] See also Figure 6 The monocular distance measuring device provided in the embodiment of the present application includes the following modules:

[0164] The target detection module 601 is used to detect the target object in the target image acquired by the image acquisition device through a deep learning algorithm to obtain a detection result;

[0165] A relationship determination module 602 is used to determine the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result;

[0166] A first construction module 603 is used to construct an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship, wherein the observation equation corresponds to a state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device;

[0167] A target value determination module 604 is used to determine a target value of a state variable based on an observation equation and a pre-constructed extended Kalman filter ranging model, wherein the extended Kalman filter ranging model corresponds to the state variable;

[0168] The distance determination module 605 is used to determine the distance between the target object and the image acquisition device according to the target value of the state variable.

[0169] In some embodiments, the detection result includes a two-dimensional target frame, a bottom edge, and a side edge corresponding to the target object, and the relationship determination module 602 is used to:

[0170] Determine the distance relationship between the target object and the image acquisition device according to the two-dimensional target frame;

[0171] The orientation relationship between the target object and the image acquisition device is determined according to the number of side edges.

[0172] In some embodiments, the first building block 603 is used to:

[0173] When it is determined that the target object is located near the image acquisition device, based on the inverse perspective transformation matrix corresponding to the image acquisition device, a first observation equation corresponding to the horizontal coordinate of the target side edge at the ground projection point, a second observation equation corresponding to the vertical coordinate of the target side edge at the ground projection point, and a third observation equation corresponding to the slope of the bottom edge are constructed, and the target side edge is the side edge detected in the target image;

[0174] The first observation equation, the second observation equation and the third observation equation are used as observation equations of the target object relative to the image acquisition device.

[0175] In some embodiments, the first building block 603 is used to:

[0176] When it is determined that the target object is far away from the image acquisition device and is not facing the image acquisition device, an observation equation corresponding to the horizontal coordinates of the three side edges in the target image is constructed to obtain three observation equations;

[0177] The three observation equations are used as the observation equations of the target object relative to the image acquisition device.

[0178] In some embodiments, the first building block 603 is used to:

[0179] When it is determined that the target object is far away from the image acquisition device and directly facing the image acquisition device, an observation equation corresponding to the horizontal coordinates of two side edges in the target image is constructed to obtain two observation equations;

[0180] Construct a constraint equation, where the constraint equation is used to indicate that the bottom edge corresponding to the orientation of the target vehicle and the optical center of the image acquisition device satisfy orthogonality;

[0181] The two observation equations and the constraint equation are used as observation equations of the target object relative to the image acquisition device.

[0182] In some embodiments, the target value determination module 604 is specifically configured to:

[0183] Determine the first Jacobian matrix of the observation equation;

[0184] Obtaining a second Jacobian matrix of a motion equation of the target object relative to the image acquisition device, the motion equation corresponding to the state variable;

[0185] The first Jacobian matrix and the second Jacobian matrix are input into the extended Kalman filter ranging model, and the prediction model in the extended Kalman filter ranging model is used to predict the state variable according to the first Jacobian matrix to obtain the predicted value of the state variable, and the update model in the extended Kalman filter ranging model is used to update the predicted value according to the second Jacobian matrix to obtain the target value of the state variable.

[0186] In some embodiments, state variables may include:

[0187] Position coordinates, velocity scalar, heading angle scalar, rotation angular velocity, width and length.

[0188] In some embodiments, the apparatus further comprises a second building block for:

[0189] Before obtaining the second Jacobian matrix of the motion equation of the target object relative to the image acquisition device, the motion equation of the target object relative to the image acquisition device is constructed based on the assumption that the target object moves in the assumed xoy plane and satisfies the constant speed and constant angular velocity.

[0190] In some embodiments, the apparatus further comprises a third building block for:

[0191] Before inputting the first Jacobian matrix and the second Jacobian matrix into the pre-constructed extended Kalman filter ranging model, constructing an extended Kalman filter ranging model corresponding to the state variable based on the extended Kalman filter algorithm, the extended Kalman filter ranging model including a prediction model and an update model;

[0192] The prediction model is used to predict the state variables based on the motion equation of the target object relative to the image acquisition device;

[0193] The updating model is used to correct the output result of the prediction model based on the observation equation of the target object relative to the image acquisition device.

[0194] The monocular ranging device provided in the embodiment of the present application can implement each process implemented in the embodiment of the monocular ranging method. To avoid repetition, it will not be described here.

[0195] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0196] The electronic device may include a processor 301 and a memory 702 storing computer program instructions.

[0197] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0198] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 702 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory. The memory 702 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, typically, the memory 702 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it may perform the operations described in any one of the monocular ranging methods in the above-mentioned embodiments.

[0199] The processor 701 implements any one of the monocular ranging methods in the above embodiments by reading and executing the computer program instructions stored in the memory 702.

[0200] In one example, the electronic device may further include a communication interface 703 and a bus 710. Figure 7 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 and communicate with each other.

[0201] The communication interface 703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0202] Bus 710 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 710 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0203] In addition, in combination with the monocular ranging method in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the monocular ranging methods in the above embodiments is implemented.

[0204] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0205] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0206] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0207] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0208] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A monocular distance measurement method, characterized in that: The method comprises: Detecting the target object in the target image captured by the image acquisition device through a deep learning algorithm to obtain a detection result; Determine the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result; Constructing an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship, wherein the observation equation corresponds to a state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device; Determining a target value of the state variable based on the observation equation and a pre-constructed extended Kalman filter ranging model, the extended Kalman filter ranging model corresponding to the state variable; The distance between the target object and the image acquisition device is determined according to the target value of the state variable.

2. The method according to claim 1, characterized in that The detection result includes a two-dimensional target frame, a bottom edge, and a side edge corresponding to the target object, and determining the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result includes: Determine the distance relationship between the target object and the image acquisition device according to the two-dimensional target frame; The orientation relationship between the target object and the image acquisition device is determined according to the number of the side edges.

3. The method according to claim 2, characterized in that The step of constructing an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship includes: When it is determined that the target object is located near the image acquisition device, based on the inverse perspective transformation matrix corresponding to the image acquisition device, a first observation equation corresponding to the horizontal coordinate of the target side edge at the ground projection point, a second observation equation corresponding to the vertical coordinate of the target side edge at the ground projection point, and a third observation equation corresponding to the slope of the bottom edge are constructed, wherein the target side edge is the side edge detected in the target image; The first observation equation, the second observation equation and the third observation equation are used as observation equations of the target object relative to the image acquisition device.

4. The method according to claim 2, characterized in that: The step of constructing an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship includes: When it is determined that the target object is far away from the image acquisition device and is not facing the image acquisition device, construct observation equations corresponding to the horizontal coordinates of three side edges in the target image to obtain three observation equations; The three observation equations are used as observation equations of the target object relative to the image acquisition device.

5. The method according to claim 2, characterized in that: The step of constructing an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship includes: When it is determined that the target object is located far away from the image acquisition device and directly facing the image acquisition device, construct observation equations corresponding to the horizontal coordinates of two side edges of the side edges in the target image to obtain two observation equations; Constructing a constraint equation, wherein the constraint equation is used to indicate that the bottom edge corresponding to the orientation of the target vehicle and the optical center of the image acquisition device satisfy orthogonality; The two observation equations and the constraint equation are used as observation equations of the target object relative to the image acquisition device.

6. The method according to claim 1, characterized in that Determining the target value of the state variable based on the observation equation and the pre-constructed extended Kalman filter ranging model includes: determining a first Jacobian matrix of the observation equation; Acquire a second Jacobian matrix of a motion equation of the target object relative to the image acquisition device, wherein the motion equation corresponds to the state variable; The first Jacobian matrix and the second Jacobian matrix are input into the extended Kalman filter ranging model, the state variable is predicted according to the first Jacobian matrix using the prediction model in the extended Kalman filter ranging model to obtain the predicted value of the state variable, and the predicted value is updated according to the second Jacobian matrix using the update model in the extended Kalman filter ranging model to obtain the target value of the state variable.

7. The method according to any one of claims 1 to 6, characterized in that: The state variables include: Position coordinates, velocity scalar, heading angle scalar, rotation angular velocity, width and length.

8. The method according to claim 6, characterized in that Before obtaining the second Jacobian matrix of the motion equation of the target object relative to the image acquisition device, the method further includes: Based on the assumption that the target object moves in the xoy plane and satisfies the constant velocity and constant angular velocity, a motion equation of the target object relative to the image acquisition device is constructed.

9. The method according to claim 6, characterized in that Before inputting the first Jacobian matrix and the second Jacobian matrix into a pre-built extended Kalman filter ranging model, the method further includes: Constructing the extended Kalman filter ranging model corresponding to the state variable based on the extended Kalman filter algorithm, wherein the extended Kalman filter ranging model includes a prediction model and an update model; The prediction model is used to predict the state variable based on the motion equation of the target object relative to the image acquisition device; The update model is used to correct the result output by the prediction model based on the observation equation of the target object relative to the image acquisition device.

10. A monocular distance measuring device, characterized in that: The device comprises: The target detection module is used to detect the target object in the target image acquired by the image acquisition device through a deep learning algorithm to obtain a detection result; A relationship determination module, used to determine the distance relationship and the orientation relationship between the target object and the image acquisition device according to the detection result; A first construction module is used to construct an observation equation of the target object relative to the image acquisition device according to the distance relationship and the orientation relationship, wherein the observation equation corresponds to a state variable of the target object, and the state variable is related to the distance between the target object and the image acquisition device; A target value determination module, configured to determine a target value of the state variable based on the observation equation and a pre-constructed extended Kalman filter ranging model, wherein the extended Kalman filter ranging model corresponds to the state variable; A distance determination module is used to determine the distance between the target object and the image acquisition device according to the target value of the state variable.

11. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the monocular ranging method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the monocular ranging method according to any one of claims 1 to 9 is implemented.

Images