Method and apparatus for determining position of a rod, electronic device and medium

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for determining the position of a rod-shaped object. The method includes: acquiring a first image sequence and a corresponding odometer information sequence, wherein the first image sequence includes multiple frames of first images, and the odometer information sequence includes odometer poses corresponding to each first image; determining target rod-shaped object information included in each first image based on the first image sequence; determining target Plück coordinates of the target rod-shaped object based on the target rod-shaped object information included in each first image and the odometer information sequence; and determining the position of the target rod-shaped object based on the target Plück coordinates. The rod-shaped object position determined by this disclosure based on Plück coordinates is less susceptible to occlusion, interference, or detection failure in image detection results, thus significantly improving the accuracy of rod-shaped object position reconstruction.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method, apparatus, electronic device, and medium for determining the position of a rod-shaped object. Background Technology

[0002] In environmental map reconstruction scenarios, there are often pole-like objects, such as vertical poles like streetlights and utility poles, and horizontal poles like gantries, barriers, and height restriction bars, requiring location reconstruction of these objects. Related technologies typically use multiple frames of images with a certain temporal sequence to detect and track the endpoints of the poles. Then, based on the two endpoints observed in each frame, a least-squares nonlinear optimization problem is constructed to solve for the spatial coordinates (e.g., 3D coordinates in the world coordinate system) of the endpoints. However, this method of reconstructing pole locations is vulnerable when the detection results are occluded, interfered with, or fail to be detected. The tracked objects between different temporal frames cannot be guaranteed to be physically identical points, which can easily cause the state variables to tend towards these outliers during the nonlinear optimization process, resulting in poor accuracy in pole location reconstruction. Summary of the Invention

[0003] To address the aforementioned technical problems, such as poor accuracy in 3D reconstruction of rod-shaped objects, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, electronic device, and medium for determining the position of a rod-shaped object.

[0004] According to one aspect of the present disclosure, a method for determining the position of a rod-shaped object is provided, comprising: acquiring a first image sequence and an odometer information sequence corresponding to the first image sequence, the first image sequence including multiple frames of first images, and the odometer information sequence including an odometer pose corresponding to each of the first images; determining target rod-shaped object information included in each of the first images based on the first image sequence; determining target Plück coordinates of the target rod-shaped object based on the target rod-shaped object information included in each of the first images and the odometer information sequence; and determining the position of the target rod-shaped object based on the target Plück coordinates.

[0005] According to another aspect of the present disclosure, an apparatus for determining the position of a rod-shaped object is provided, comprising: a first acquisition module, configured to acquire a first image sequence and an odometer information sequence corresponding to the first image sequence, the first image sequence comprising multiple frames of first images, and the odometer information sequence comprising an odometer pose corresponding to each of the first images; a first processing module, configured to determine target rod-shaped object information included in each of the first images based on the first image sequence; a second processing module, configured to determine target Plück coordinates of the target rod-shaped object based on the target rod-shaped object information included in each of the first images and the odometer information sequence; and a third processing module, configured to determine the position of the target rod-shaped object based on the target Plück coordinates.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the method for determining the position of a rod-shaped object as described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method for determining the position of a rod-shaped object as described in any of the above embodiments of the present disclosure.

[0008] Based on the method, apparatus, electronic device, and medium for determining the position of a rod-shaped object provided in the above embodiments of this disclosure, by determining the Plück coordinates of the rod-shaped object, the straight line on which the rod-shaped object lies can be determined, and then the position of the rod-shaped object can be determined based on the Plück coordinates, thereby realizing the position reconstruction of the rod-shaped object. Since the Plück coordinates represent the straight line information of the rod-shaped object, compared with the related technology that determines the position of the rod-shaped object based on the endpoint of the rod-shaped object, the position of the rod-shaped object determined by this disclosure is not easily affected by the image detection results being occluded, interfered with, or failing to detect, etc., and therefore can greatly improve the reconstruction accuracy of the position of the rod-shaped object.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1This is an exemplary application scenario of the method for determining the position of the rod-shaped object provided in this disclosure;

[0012] Figure 2 This is a flowchart illustrating a method for determining the position of a rod-shaped object according to an exemplary embodiment of this disclosure;

[0013] Figure 3 This is a flowchart illustrating a method for determining the position of a rod-shaped object according to another exemplary embodiment of this disclosure;

[0014] Figure 4 This is a flowchart illustrating step 2033 provided in an exemplary embodiment of this disclosure;

[0015] Figure 5 This is a flowchart illustrating step 2033 provided in another exemplary embodiment of this disclosure;

[0016] Figure 6 This is a schematic diagram of a first plane and a second plane provided in an exemplary embodiment of this disclosure;

[0017] Figure 7 This is a schematic diagram illustrating the principle of determining the three-dimensional endpoint coordinates of the two ends of a target rod-shaped object according to an exemplary embodiment of this disclosure;

[0018] Figure 8 This is a schematic diagram of the structure of a device for determining the position of a rod-shaped object provided in an exemplary embodiment of this disclosure;

[0019] Figure 9 This is a schematic diagram of the structure of a device for determining the position of a rod-shaped object provided in another exemplary embodiment of this disclosure:

[0020] Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0021] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0022] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0023] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0024] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0025] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0026] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0027] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0028] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0029] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0030] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0031] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0032] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0033] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0034] This disclosure outlines

[0035] In the process of developing this disclosure, the inventors discovered that in environmental map reconstruction scenarios, there are often some pole-like objects, such as vertical poles like streetlights and utility poles, and horizontal poles like gantry frames, barriers, and height restriction poles, requiring position reconstruction of these pole-like objects. Related technologies typically use multiple frames of images with a certain temporal sequence to detect and track the endpoints of the pole-like objects, and then construct a least-squares nonlinear optimization problem based on the two endpoints observed in each frame to solve for the spatial coordinates (e.g., three-dimensional coordinates in the world coordinate system) of the two endpoints. However, this method of reconstructing the position of pole-like objects has limitations. When the detection results of the pole-like objects are occluded, interfered with, or fail to be detected, the tracked objects between different temporal frames cannot be guaranteed to be corresponding points in physical space. This can easily lead to the state variables approaching these outliers during the nonlinear optimization process, resulting in poor accuracy in reconstructing the position of the pole-like objects.

[0036] Exemplary Overview

[0037] Figure 1 This is an exemplary application scenario of the method for determining the position of the rod-shaped object provided in this disclosure.

[0038] In assisted driving scenarios, it is usually necessary to reconstruct the environmental map of the vehicle's surroundings. For pole-shaped objects such as streetlights, utility poles, gantries, barriers, and height restriction poles in the environment, the position determination method of the pole-shaped objects disclosed herein can be used to reconstruct the position of the pole-shaped objects, thereby improving the accuracy of the position reconstruction. Specifically, a first image sequence can be acquired based on an image sensor (such as a camera) on the vehicle, and an odometer information sequence corresponding to the first image sequence can be acquired based on an odometer on the vehicle. The first image sequence may include multiple first images, and the odometer information sequence may include the odometer pose corresponding to each first image. Target rod information included in each first image is determined based on each first image. Target Plück coordinates of the target rod are determined based on the target rod information and the odometer information sequence included in each first image. The number of target rods can be at least one. For each target rod, its corresponding target Plück coordinates can be determined, and then the position of the target rod is determined based on the target Plück coordinates. Since Plück coordinates represent the straight line information of the rod, compared with related technologies that determine the position of the rod based on the endpoint of the rod, the position of the rod determined by this disclosure is less susceptible to the influence of image detection results being occluded, interfered with, or failing to detect, thus greatly improving the reconstruction accuracy of the rod's position.

[0039] Exemplary methods

[0040] Figure 2 This is a flowchart illustrating a method for determining the position of a rod-shaped object according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 2 As shown, it includes the following steps:

[0041] Step 201: Obtain the first image sequence and the odometry information sequence corresponding to the first image sequence. The first image sequence includes multiple first images, and the odometry information sequence includes the odometry pose corresponding to each first image.

[0042] The first image sequence can be obtained based on an image sensor, such as acquiring images captured by a camera on the vehicle at various times within a time series. Specifically, it can be that each frame of the first image captured by the camera is stored, and when the position of the rod needs to be reconstructed, a first image sequence including multiple frames of the first image is acquired. These multiple frames of the first image may include the current frame and a first number of historical frames of the first image, which can be set according to actual needs. The odometer pose can be the vehicle pose in a physical space coordinate system (such as the world coordinate system) determined by combining the data collected by the odometer with the motion model. The first image sequence and the odometer information sequence can be synchronized and aligned based on the acquisition time to determine the correspondence between each first image and the odometer pose.

[0043] In one optional example, the first image sequence may include images that all contain the rod-shaped object, or images that partially contain the rod-shaped object, depending on the actual needs. For example, for each captured image frame, rod-shaped object detection is used to determine whether a rod-shaped object is observed in the image and this is recorded. Then, based on the recorded information, multiple consecutive frames of images in which the rod-shaped object is observed can be obtained as the first image sequence. Alternatively, during vehicle movement, at the current moment, the current frame and a certain number of historical frames can be acquired as the first image sequence, and subsequent processes can determine whether the rod-shaped object is included and perform further processing.

[0044] Step 202: Based on the first image sequence, determine the target rod-shaped object information included in each of the first images.

[0045] For any given first image, the target rod information may include one or more target rods, and the correspondence between target rods in each first image can be determined through target tracking between adjacent first images. The target rod information may include identification information corresponding to the target rod. This identification information is used to identify different target rods, as well as the same target rod in different first images; that is, the same target rod observed in different first images uses the same identification information. The specific representation of the identification information can be set according to actual needs, such as numbering, and is not specifically limited.

[0046] In an optional example, the target rod information included in each first image can be determined by rod target detection or semantic segmentation. For example, the target rod information obtained by target detection may include the detection box information of the detected target rod, and the target rod information obtained by semantic segmentation may include the pixel set belonging to the target rod. Alternatively, the target rod information may include both the detection result and the semantic segmentation result, which can be set according to actual needs.

[0047] Step 203: Based on the target rod information and odometer information sequence included in each of the first images, determine the target Plück coordinates of the target rod.

[0048] Among them, the target Plück coordinates are the straight line representation of the target rod in physical space, which can be obtained through a certain optimization algorithm. The specific optimization algorithm can be set according to actual needs. For example, the rod can be represented by Plück coordinates as a state variable. The state variable can be iteratively updated by observing the rod in each of the first images in the first image sequence to achieve optimization and finally obtain the target Plück coordinates.

[0049] Step 204: Determine the position of the target rod based on the target Plück coordinates.

[0050] Since the target Plück coordinates are a straight line representation of the target rod in physical space, the position of the target rod can be determined by converting the target Plück coordinates to the position coordinates in physical space.

[0051] The method for determining the position of a rod-shaped object provided in this embodiment can determine the straight line on which the rod-shaped object lies by determining the Plück coordinates of the rod-shaped object, and then determine the position of the rod-shaped object based on the Plück coordinates, thereby realizing the position reconstruction of the rod-shaped object. Since the Plück coordinates represent the straight line information of the rod-shaped object, compared with related technologies that determine the position of the rod-shaped object based on the endpoints of the rod-shaped object, the position of the rod-shaped object determined by this disclosure is not easily affected by the image detection results being occluded, interfered with, or the detection failure, etc., and thus can greatly improve the reconstruction accuracy of the position of the rod-shaped object.

[0052] Figure 3 This is a flowchart illustrating a method for determining the position of a rod-shaped object according to another exemplary embodiment of this disclosure.

[0053] In an optional embodiment, step 203 may specifically include the following steps:

[0054] Step 2031: Determine the initial Plück coordinates of the target rod based on any two frames of the first image that include the target rod and the odometry poses corresponding to the two frames of the first image.

[0055] Specifically, any two first images containing the target rod can be selected from the first image sequence according to certain rules. The specific selection rules can be set according to actual needs. For example, if the target rod is observed in multiple consecutive first images, two first images with a certain frame interval can be selected from the multiple consecutive frames. The initial Plück coordinates are obtained based on the selected two first images containing the target rod and the odometry poses corresponding to the two first images. The specific initialization method can be set according to actual needs. For example, for any one of the two first images, the pixel points corresponding to the two ends of the target rod are determined based on the target rod information corresponding to the first image. Based on the position of the two pixels in the camera coordinate system and the origin of the camera coordinate system (i.e., the observation point), a plane can be determined, which is the plane where the light path of the rod imaging is located. For the two first images, the planes corresponding to the two frames can be obtained respectively. According to the imaging principle, the intersection line of the planes of the imaging path of the same rod in the two images is the straight line of the rod. Based on this, according to the odometry poses corresponding to the two frames respectively, the planes corresponding to the two frames are transformed to the physical space coordinate system, and the intersection line of the two planes in the physical space coordinate system is determined. This intersection line is the straight line of the target rod in the physical space coordinate system. Then, the initial Plück coordinates can be determined through matrix decomposition.

[0056] For example, the initial Plück coordinates are represented as L0. w =[n0 w v0 w ] T , among which, L0 w Let n0 be a 6×1 vector. w Indicates that it contains the line L0 w The normal vector of the plane π, v0 w Represents line L0 w The direction vector in the plane π. w and v0 w All are three-dimensional vectors in physical space coordinate system, such as n0 w =[n0 w1 n0 w2 n0 w3 ], v0 w =[v0 w1 v0 w2 v0 w3 ]. Or n0 w =[n0 w1 n0 w2 n0 w3 ] T v0 w =[v0 w1 , v. w2 v0 w3 ] T Then L0 w =[n0 w T v0 w T ] T The specific representation is not limited. T represents transpose.

[0057] Step 2032: Determine the initial state variables based on the initial Plück coordinates.

[0058] Once the initial Plück coordinates are determined, the initial state variables can be determined based on these coordinates. For example, the coordinate components of the initial Plück coordinates can be used as the state components of the initial state variables. Alternatively, the initial Plück coordinates can be transformed to use them as the initial state variables, such as converting them into orthogonal state variables. The specific settings can be configured according to actual needs.

[0059] Step 2033: Based on the target rod information included in each first image, the initial state variables are iteratively updated using a preset optimization algorithm to obtain the target Plück coordinates of the target rod.

[0060] The preset optimization algorithm can be set according to actual needs. For example, a nonlinear least squares optimization algorithm can be used, and the solution can be obtained through algorithms such as Gauss-Newton and Levenberg-Marquardt to achieve iterative update of state variables.

[0061] This embodiment uses the Plück coordinates of the rod as a state variable and iteratively updates and optimizes the state variable through a certain optimization algorithm to obtain the target Plück coordinates of the target rod. This can further improve the accuracy of the Plück coordinates of the target rod, thereby improving the accuracy of the target rod's position.

[0062] Figure 4 This is a flowchart illustrating step 2033 provided in an exemplary embodiment of this disclosure.

[0063] In an optional embodiment, step 2033, based on the target rod information included in each of the first images, iteratively updates the initial state variables using a preset optimization algorithm to obtain the target Plück coordinates of the target rod, includes:

[0064] Step 20331: Based on the target rod information included in each first image, the initial state variables are iteratively updated using the Gauss-Newton method until the updated state variables meet the preset conditions and / or reach the maximum number of iterations, and then the iteration process ends.

[0065] The preset conditions and maximum number of iterations can be set according to actual needs. For example, the preset condition can be that the objective function value corresponding to the updated state variable is less than the residual threshold, without specific limitations. The objective function value can be determined based on the preset objective function, which can be constructed based on the reprojection residual of the state variable. In the Gauss-Newton method, during the iteration process, for the k-th iteration, the current Jacobian matrix and error (in this disclosure, the reprojection residual of the state variable) are determined. Based on the current Jacobian matrix and error, the incremental equation is determined. By solving the incremental equation, the current iteration step size is obtained, which is used to update the state variable obtained in the previous iteration, thereby realizing the iterative update of the state variable.

[0066] Step 20332: Based on the updated state variables, determine the target Plück coordinates of the target rod.

[0067] Since the state variables are determined based on Plück coordinates and include all components of the Plück coordinates, the updated state variables can be extracted to obtain the target Plück coordinates.

[0068] This embodiment uses the Gauss-Newton method to solve for iterative updates of state variables, which can improve the iteration convergence speed.

[0069] Figure 5This is a flowchart illustrating step 2033 provided in another exemplary embodiment of this disclosure.

[0070] In an optional embodiment, step 20331, based on the target rod information included in each first image, iteratively updates the initial state variables using the Gauss-Newton method until the updated state variables meet preset conditions and / or reach the maximum number of iterations, ending the iteration process, includes:

[0071] 1. During the iteration process, for the current iteration, based on the target rod information included in each first image, the odometry pose corresponding to each first image, and the first state quantity obtained in the previous iteration, the current reprojection residual and the current Jacobian matrix corresponding to the first state quantity are determined. The current reprojection residual includes the reprojection residual of the first state quantity on each first image, and the current Jacobian matrix includes the Jacobian matrix of the current reprojection residual with respect to the first state quantity. When the current iteration is the first iteration, the first state quantity is the initial state quantity.

[0072] The reprojection residual of the first state variable on any first image can be represented by the distance between the projection line (or line segment) of the first state variable on that first image and the two endpoints of the target rod included in that first image (which can be determined based on the target rod information). The smaller the distance, the smaller the reprojection residual, meaning that the Plück coordinates corresponding to the first state variable are closer to the actual target rod. The reprojection residuals of the first state variable on each first image are used as the current reprojection residuals for subsequent iterations, thereby making the Plück coordinates corresponding to the state variables obtained through iteration increasingly closer to the actual target rod. The current Jacobian matrix can be obtained by differentiation using the chain rule. The current Jacobian matrix represents the gradient direction of the iteration.

[0073] For example, taking the first state variable as an orthogonal state variable, for the j-th frame of the first image, the reprojection residual of the first state variable in the first image is represented as e. Lj The Jacobian matrix of the reprojection residual with respect to the first state variable can be expressed as:

[0074]

[0075] in, It also means e Lj Taking the partial derivative with respect to the first state variable, e Lj Regarding the Pluke coordinates L w Jacobian (partial derivative) of the projected line l' in the first image of the j-th frame; This represents the line L' projected onto the camera coordinate system corresponding to the first image in the j-th frame. c Jacobi; The line L in the camera coordinate system c Regarding the straight line L in the world coordinate system w Jacobi; L represents w Regarding the Jacobian of orthogonal state variables (first state variables); the orthogonal state variables and Plück coordinates have a certain transformation relationship, which can be expressed as follows:

[0076]

[0077] Where n represents the line L w The normal vector of the plane, v represents the line L. w The direction vector is ||||, which represents finding the magnitude of the vector, and [U, W] represents orthogonal state variables. U = [u1, u2, u3], where u1, u2, and u3 represent the 1st, 2nd, and 3rd columns of U, respectively.

[0078] When the Plück coordinates are used as state variables, the current Jacobian matrix is ​​also the Jacobian matrix of the current reprojection residual with respect to the orthogonal state variables, which is determined by the chain rule mentioned above. The Plück coordinates are then optimized through orthogonal optimization.

[0079] The above describes the reprojection residual corresponding to a first frame of image. The current reprojection residual and the current Jacobian matrix both include each first image. Therefore, the resulting current Jacobian matrix includes the Jacobian corresponding to each first frame of image, and is a large Jacobian matrix.

[0080] 2. Based on the current reprojection residual and the current Jacobian matrix corresponding to the first state variable, determine the current step size corresponding to the current iteration.

[0081] The current iteration step size can be obtained by solving the relational equation between the current Jacobian matrix, the current step size, and the current reprojection residual.

[0082] For example, the current Jacobian matrix is ​​denoted as J(x), the current step size is denoted as Δx, and the current reprojection residual is denoted as f(x), where x represents the state variable. In the Gauss-Newton method, the following system of equations is satisfied:

[0083] J(x) T J(x)Δx=-J(x) T f(x)

[0084] Here, J(x) and f(x) are known quantities, and the current step size Δx can be obtained by solving this system of equations.

[0085] 3. Based on the current step size and the first state variable, determine the updated second state variable.

[0086] After obtaining the current step size, the first state variable can be updated based on the current step size to obtain the updated second state variable.

[0087] 4. The iteration process ends when the reprojection residual corresponding to the second state variable meets the preset conditions and / or the current iteration reaches the maximum number of iterations.

[0088] For specific instructions on this step, please refer to the preceding content.

[0089] In an optional example, the end of the iteration process can also be determined based on the iteration step size. For example, when the determined current step size is small enough (less than the step size threshold), it means that the state variable has been updated to meet certain conditions, and then the iteration process ends.

[0090] Step 20332, based on the updated state variables, determines the target Plück coordinates of the target rod, including:

[0091] a. Determine the target Plück coordinates of the target rod based on the second state variable.

[0092] For specific instructions on this step, please refer to the steps outlined above.

[0093] This embodiment determines the step size of the current iteration for iterative updates of the state variables by using the reprojection residuals from the state variables to each first image and the Jacobian matrix of the reprojection residuals with respect to the first state variables. This continuously optimizes the Plück coordinates of the target rod and further improves the accuracy of the target rod's position.

[0094] In an optional embodiment, the target rod information includes the pixel coordinates of the two endpoints of the target rod in the first image; based on the target rod information included in each first image, the odometry pose corresponding to each first image, and the first state variable obtained in the previous iteration, the current reprojection residual corresponding to the first state variable is determined, including:

[0095] For each frame of the first image, based on the odometry pose and the first state quantity corresponding to the first image, a first intermediate quantity in the camera coordinate system corresponding to the first state quantity is determined; based on the first intermediate quantity, the projection line of the first state quantity on the first image is determined; based on the projection line and the pixel coordinates of the two ends of the target rod in the first image, the reprojection residual of the first state quantity in the first image is determined; based on the reprojection residual of the first state quantity in each first image, the current reprojection residual is determined.

[0096] The first intermediate quantity is obtained by transforming the first state quantity to the camera coordinate system corresponding to the first image. This transformation can be based on the camera pose corresponding to the first image, which can be determined based on the relative relationship between the camera and vehicle poses, and the odometer pose corresponding to the first image. The first intermediate quantity represents the Plücker coordinate line (L) corresponding to the first state quantity. w The straight line (L) corresponding to the camera coordinate system c The projection line of the first state variable onto the first image is the projection line (l′) on the image coordinate system obtained by transforming the first intermediate variable back to the image coordinate system corresponding to the first image. The principle of the transformation from the camera coordinate system to the image coordinate system will not be elaborated here. The reprojection residual of the first state variable onto the first image can be represented by the distance between the pixel coordinates of the two ends of the target rod in the first image and the projection line. The principle of determining the distance from the point to the line will not be elaborated here.

[0097] For example, the first state variable represented by Plück coordinates is L. w =[n w v w ] T The camera intrinsic parameter corresponding to the first image is f. x f y c x c y The camera pose (external parameters) is [R] cw , t cw ], R cw Indicates camera rotation amount, t cw L represents the camera translation amount. w Transform to camera coordinate system to obtain L c =[n c v c ] T , means as follows:

[0098]

[0099] Among them, [t cw ] × Indicates t cw An antisymmetric matrix. 6×6 and 6×1 represent the matrix dimensions.

[0100] Take the first intermediate quantity L c The projected line obtained by projecting it onto the first image is l' = [l1, l2, l3]. T , means as follows:

[0101]

[0102] Assuming this first image is the first image of the j-th frame, the pixel coordinates of the two ends of the target rod in this first image are represented as pa=(u a v a ) and pb=(u b v b The first state variable is the reprojection residual e on the first image in the j-th frame. Lj Represented as:

[0103]

[0104] The objective function is constructed as follows:

[0105]

[0106] Where N represents the total number of first images included in the first image sequence, w j The weights represent empirical values. The goal of state optimization is to minimize the objective function F.

[0107] In this embodiment, the reprojection residual is represented by the distance between the pixel coordinates of the two ends of the target rod detected in the first image and the projection line of the first state quantity in the first image. This distance is used to determine the current reprojection residual so as to facilitate the updating of the Plück coordinates of the target rod.

[0108] In an optional embodiment, based on the target rod information included in each first image and the first state quantity obtained in the previous iteration, the current Jacobian matrix corresponding to the first state quantity is determined, including:

[0109] Based on the pixel coordinates of the two endpoints of the projected line and the target rod in the first image, the first Jacobian of the reprojection residual with respect to the projected line is determined; based on the projection matrix from the first intermediate quantity to the projected line, the second Jacobian of the projected line with respect to the first intermediate quantity is determined; based on the transformation matrix from the first state quantity to the camera coordinate system, the third Jacobian of the first intermediate quantity with respect to the first state quantity is determined; based on the transformation relationship between the first state quantity and the orthogonal state quantity, the fourth Jacobian of the first state quantity with respect to the orthogonal state quantity is determined; based on the first, second, third, and fourth Jacobian, the fifth Jacobian of the first state quantity corresponding to the first image is determined; based on the fifth Jacobian of the first state quantity corresponding to each of the first images, the current Jacobian matrix is ​​determined.

[0110] The first Jacobi is the one mentioned above. It is expressed as follows:

[0111]

[0112] The second Jacobi is the one mentioned above. It is expressed as follows:

[0113]

[0114] The third Jacobi is as mentioned above. It is expressed as follows:

[0115]

[0116] The fourth Jacobi is It is expressed as follows:

[0117]

[0118] The fifth Jacobi is It is expressed as follows:

[0119]

[0120] The meanings of the above symbols are explained in the foregoing content and will not be repeated here.

[0121] In an optional embodiment, the current step size corresponding to the current iteration is determined based on the current reprojection residual corresponding to the first state variable and the current Jacobian matrix, including:

[0122] Based on the current Jacobian matrix, determine the Hessian matrix; based on the Hessian matrix, the current Jacobian matrix, and the current reprojection residual, determine the normal equation regarding the iteration step size; based on the normal equation, determine the current step size.

[0123] Here, the current Jacobian matrix is ​​denoted as J(x), and the Hessian matrix is ​​denoted as J(x). T The normal equation for J(x) is:

[0124] J(x) T J(x)Δx=-J(x) T f(x)

[0125] The current step size Δx can be obtained by solving this system of equations.

[0126] This embodiment determines the current step size using the Hessian matrix, which avoids the calculation of the second derivative, effectively reducing the amount of computation and improving processing efficiency.

[0127] In an optional embodiment, step 202, based on the first image sequence, determines the target rod-shaped object information included in each of the first images, including:

[0128] Step 2021: For each frame of the first image in the first image sequence, a pre-trained rod detection model is used to determine the pixel region corresponding to the rod included in the first image.

[0129] The rod-shaped object detection model can be any feasible object detection model or semantic segmentation model, and this disclosure does not impose any limitations. For example, the object detection model can be a YOLO series model, and the semantic segmentation model can be a model based on the U-Net (U-shaped network) series, a model based on FCN (Fully Convolutional Networks), a model based on the DeepLab series, and so on.

[0130] Step 2022: Based on the pixel regions corresponding to the rod-shaped objects included in each first image, a preset target tracking algorithm is used to track the target rod-shaped objects in the rod-shaped objects, and the target rod-shaped object information included in each first image is determined.

[0131] The preset target tracking algorithm can be any feasible algorithm, such as optical flow, Kalman filtering, particle filtering, etc. The specific algorithm can be set according to actual needs.

[0132] This embodiment uses rod detection and tracking to match rods belonging to the same target in each first image, providing accurate target rod information for subsequent optimization of Plück coordinates.

[0133] In an optional embodiment, step 204, determining the position of the target rod based on the target Plück coordinates, includes:

[0134] Step 2041: Based on the target rod information included in each first image and the preset rules, determine the first target image and the second target image from each first image.

[0135] The preset rules can be set according to actual needs, such as randomly selecting two frames from multiple first images that include the same target rod, selecting two frames at a certain interval, and so on.

[0136] In an optional example, to further improve the accuracy of the target rod's location, the two frames with the best observation effect can be selected from the first images that include the same target rod. For example, the two first images with the longest observed target rod can be selected as the first target image and the second target image, respectively.

[0137] Step 2042: Based on the first camera optical center coordinates in the first camera coordinate system corresponding to the first target image, the second camera optical center coordinates in the first camera coordinate system corresponding to the second target image, and the first target endpoint and the second target endpoint in the first camera coordinate system corresponding to the first endpoint and the second target endpoint of the target rod on the first target image, respectively, determine the first plane and the second plane in the first camera coordinate system.

[0138] The second camera optical center coordinates are coordinates that transform the camera optical center coordinates corresponding to the second target image from the second camera coordinate system corresponding to the second target image to the first camera coordinate system of the first target image, unifying the optical centers of the two frames to the same coordinate system. The three points of the first target endpoint, the first camera optical center coordinates, and the second camera optical center coordinates can determine a plane (as the first plane), and the second target endpoint, the first camera optical center coordinates, and the second camera optical center coordinates can determine another plane (as the second plane).

[0139] For example, Figure 6 This is a schematic diagram of a first plane and a second plane provided in an exemplary embodiment of this disclosure. C1 represents the optical center coordinates of the first camera, and C2 represents the optical center coordinates of the second camera. The first endpoint and the second endpoint on the first target image are transformed into the first camera coordinate system to obtain the first target endpoint and the second target endpoint in the first camera coordinate system. The first target endpoint, C1, and C2 form the first plane, and the second target endpoint, C1, and C2 form the second plane.

[0140] Step 2043: Transform the first plane and the second plane to the world coordinate system to obtain the third plane and the fourth plane in the world coordinate system.

[0141] The transformation from the first camera coordinate system to the world coordinate system can be achieved based on the extrinsic parameters of the first camera; the specific principle will not be elaborated here.

[0142] In an optional embodiment, the optical center coordinates of the first camera corresponding to the first image and the optical center coordinates of the second image can both be transformed to the world coordinate system, and the first target endpoint and the second target endpoint can also be transformed to the world coordinate system, so that the third plane and the fourth plane in the world coordinate system can be directly obtained.

[0143] Step 2044: Based on the third plane, the fourth plane, and the target Plück coordinates, determine the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system.

[0144] The target Plück coordinates represent the straight line of the target rod in the world coordinate system. The intersection of the straight line based on the target Plück coordinates and the point plane can determine the three-dimensional endpoint coordinates of one end of the target rod. The intersection of the straight line based on the target Plück coordinates and the fourth plane can determine the three-dimensional endpoint coordinates of the other end of the target rod.

[0145] For example, Figure 7 This is a schematic diagram illustrating the principle of determining the three-dimensional endpoint coordinates of the two ends of a target rod-shaped object according to an exemplary embodiment of this disclosure. Figure 6Based on this, the first and second planes are transformed to the world coordinate system to obtain the third and fourth planes. The target Plücker coordinates represent the line as L = [n, v]. T Let C be the intersection of the third plane and L, and D be the intersection of the fourth plane and L. Then C and D are the three-dimensional endpoint coordinates of the two ends of the target rod.

[0146] Step 2045: Determine the three-dimensional coordinates of the target rod based on the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system.

[0147] The three-dimensional coordinates of the target rod can include the coordinates of its two endpoints in the world coordinate system. Alternatively, the three-dimensional coordinates of the target rod can be represented in other ways, such as using line segments, depending on the specific requirements.

[0148] Step 2046: Use the three-dimensional coordinates as the position of the target rod.

[0149] This embodiment selects the optimal first image through preset rules to determine the first and second planes, and then transforms them to the world coordinate system to obtain the third and fourth planes. Based on the intersection of the third and fourth planes with the spatial line represented by the target Plück coordinates, the three-dimensional endpoint coordinates of the two ends of the target rod are determined, effectively improving the accuracy of the three-dimensional endpoint coordinates.

[0150] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0151] The method for determining the position of any rod-shaped object provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the method for determining the position of any rod-shaped object provided in this disclosure can be executed by a processor, such as by a processor calling corresponding instructions stored in memory to execute the method for determining the position of any rod-shaped object mentioned in this disclosure. Further details will not be elaborated below.

[0152] Exemplary device

[0153] Figure 8 This is a schematic diagram of a device for determining the position of a rod-shaped object according to an exemplary embodiment of this disclosure. The device of this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 8 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0154] The first acquisition module 501 is used to acquire a first image sequence and an odometer information sequence corresponding to the first image sequence. The first image sequence includes multiple frames of first images, and the odometer information sequence includes the odometer pose corresponding to each first image.

[0155] The first processing module 502 is used to determine the target rod-shaped object information included in each of the first images based on the first image sequence;

[0156] The second processing module 503 is used to determine the target Plück coordinates of the target rod based on the target rod information and odometer information sequence included in each of the first images.

[0157] The third processing module 504 is used to determine the position of the target rod based on the target Plück coordinates.

[0158] Figure 9 This is a schematic diagram of the structure of a device for determining the position of a rod-shaped object provided in another exemplary embodiment of this disclosure.

[0159] In an optional embodiment, the second processing module 503 includes:

[0160] The first processing unit 5031 is used to determine the initial Plück coordinates of the target rod based on any two frames of first images including the target rod in each first image and the odometer poses corresponding to the two frames of first images respectively.

[0161] The second processing unit 5032 is used to determine the initial state quantities based on the initial Plück coordinates;

[0162] The third processing unit 5033 is used to iteratively update the initial state variables based on the target rod information included in each of the first images, using a preset optimization algorithm, to obtain the target Plück coordinates of the target rod.

[0163] In an optional embodiment, the third processing unit 5033 is specifically used for:

[0164] Based on the target rod information included in each first image, the initial state variables are iteratively updated using the Gauss-Newton method until the updated state variables meet the preset conditions and / or reach the maximum number of iterations, at which point the iteration process ends; based on the updated state variables, the target Plück coordinates of the target rod are determined.

[0165] In an optional embodiment, the third processing unit 5033 is specifically used for:

[0166] During the iteration process, for the current iteration, based on the target rod information included in each first image, the odometry pose corresponding to each first image, and the first state variable obtained in the previous iteration, the current reprojection residual and the current Jacobian matrix corresponding to the first state variable are determined. The current reprojection residual includes the reprojection residual of the first state variable on each first image, and the current Jacobian matrix includes the Jacobian matrix of the current reprojection residual with respect to the first state variable. When the current iteration is the first iteration, the first state variable is the initial state variable. Based on the current reprojection residual and the current Jacobian matrix corresponding to the first state variable, the current step size corresponding to the current iteration is determined. Based on the current step size and the first state variable, the updated second state variable is determined. In response to the reprojection residual corresponding to the second state variable satisfying the preset condition and / or the current iteration reaching the maximum number of iterations, the iteration process ends. Based on the second state variable, the target Plück coordinates of the target rod are determined.

[0167] In an optional embodiment, the target rod information includes the pixel coordinates of the two endpoints of the target rod in the first image; the third processing unit 5033 is specifically used for:

[0168] For each frame of the first image, based on the odometry pose and the first state quantity corresponding to the first image, a first intermediate quantity in the camera coordinate system corresponding to the first state quantity is determined; based on the first intermediate quantity, the projection line of the first state quantity on the first image is determined; based on the projection line and the pixel coordinates of the two ends of the target rod in the first image, the reprojection residual of the first state quantity in the first image is determined; based on the reprojection residual of the first state quantity in each first image, the current reprojection residual is determined.

[0169] In an optional embodiment, the third processing unit 5033 is specifically used for:

[0170] Based on the pixel coordinates of the two endpoints of the projected line and the target rod in the first image, the first Jacobian of the reprojection residual with respect to the projected line is determined; based on the projection matrix from the first intermediate quantity to the projected line, the second Jacobian of the projected line with respect to the first intermediate quantity is determined; based on the transformation matrix from the first state quantity to the camera coordinate system, the third Jacobian of the first intermediate quantity with respect to the first state quantity is determined; based on the transformation relationship between the first state quantity and the orthogonal state quantity, the fourth Jacobian of the first state quantity with respect to the orthogonal state quantity is determined; based on the first, second, third, and fourth Jacobian, the fifth Jacobian of the first state quantity corresponding to the first image is determined; based on the fifth Jacobian of the first state quantity corresponding to each of the first images, the current Jacobian matrix is ​​determined.

[0171] In an optional embodiment, the third processing unit 5033 is specifically used for:

[0172] Based on the current Jacobian matrix, determine the Hessian matrix; based on the Hessian matrix, the current Jacobian matrix, and the current reprojection residual, determine the normal equation regarding the iteration step size; based on the normal equation, determine the current step size.

[0173] In an optional embodiment, the first processing module 502 includes: a first determining unit 5021 and a second determining unit 5022.

[0174] The first determining unit 5021 is used to determine the pixel region corresponding to the rod-shaped object included in each frame of the first image in the first image sequence by using a pre-trained rod-shaped object detection model.

[0175] The second determining unit 5022 is used to track the target rod in the rod based on the pixel region corresponding to the rod included in each first image, using a preset target tracking algorithm, and determine the target rod information included in each first image.

[0176] In an optional embodiment, the third processing module 504 includes: a third determining unit 5041, a fourth determining unit 5042, a conversion unit 5043, a fifth determining unit 5044, a sixth determining unit 5045, and a seventh determining unit 5046.

[0177] The third determining unit 5041 is used to determine the first target image and the second target image from each first image based on the target rod information included in each first image and preset rules.

[0178] The fourth determining unit 5042 is used to determine the first plane and the second plane in the first camera coordinate system based on the first camera optical center coordinates in the first camera coordinate system corresponding to the first target image, the second camera optical center coordinates in the first camera coordinate system corresponding to the second target image, and the first target endpoint and the second target endpoint in the first camera coordinate system corresponding to the first endpoint and the second target endpoint of the target rod on the first target image, respectively.

[0179] The transformation unit 5043 is used to transform the first plane and the second plane to the world coordinate system, respectively, to obtain the third plane and the fourth plane in the world coordinate system.

[0180] The fifth determining unit 5044 is used to determine the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system based on the third plane, the fourth plane, and the target Plück coordinates.

[0181] The sixth determining unit 5045 is used to determine the three-dimensional coordinates of the target rod based on the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system.

[0182] The seventh determining unit 5046 is used to determine the position of the target rod-shaped object using three-dimensional coordinates.

[0183] The modules and units in this disclosed device can be further divided into more granular categories according to actual needs. For example, a unit can be divided into multiple sub-units, without any specific limitation.

[0184] Exemplary electronic devices

[0185] This disclosure also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the method for determining the position of the rod-shaped object as described in any of the above embodiments of this disclosure.

[0186] Figure 10 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0187] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0188] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0189] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0190] For example, the input device 13 can be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0191] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0192] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0193] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0194] Exemplary computer program products and computer-readable storage media

[0195] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0196] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0197] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0198] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0199] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0200] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0201] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0202] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0203] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0204] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0205] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for determining the position of a rod-shaped object, comprising: A first image sequence and an odometer information sequence corresponding to the first image sequence are obtained. The first image sequence includes multiple frames of first images, and the odometer information sequence includes the odometer pose corresponding to each of the first images. Based on the first image sequence, determine the target rod-shaped object information included in each of the first images; Based on the target rod information and the odometer information sequence included in each of the first images, the target Plück coordinates of the target rod are determined; The position of the target rod is determined based on the target Plück coordinates; The target Plück coordinates are obtained as follows: the target rod is represented by Plück coordinates as a state variable, and the state variable is iteratively updated based on the observation of the target rod in each of the first images in the first image sequence to obtain the target Plück coordinates; The position of the target rod is obtained as follows: based on the target rod information included in each of the first images and preset rules, a first target image and a second target image are determined from each of the first images; based on the first camera optical center corresponding to the first target image, the second camera optical center corresponding to the second target image, and the first target endpoint and the second target endpoint in the first camera coordinate system corresponding to the first endpoint and the second target endpoint of the target rod on the first target image, respectively, a third plane and a fourth plane in the world coordinate system are determined; based on the intersection of the third plane and the fourth plane with the spatial line represented by the target Plück coordinates, the position of the target rod is determined.

2. The method of claim 1, wherein, The step of determining the target Plück coordinates of the target rod based on the target rod information included in each of the first images and the odometer information sequence includes: Based on any two frames of the first image that include the target rod, and the odometer pose corresponding to the two frames of the first image, the initial Plück coordinates of the target rod are determined; Based on the initial Plück coordinates, determine the initial state quantities; Based on the target rod information included in each of the first images, the initial state variables are iteratively updated using a preset optimization algorithm to obtain the target Plück coordinates of the target rod.

3. The method according to claim 2, wherein, The step of obtaining the target Plück coordinates of the target rod based on the target rod information included in each of the first images, by iteratively updating the initial state variables using a preset optimization algorithm, includes: Based on the target rod information included in each of the first images, the initial state quantity is iteratively updated using the Gauss-Newton method until the updated state quantity meets the preset conditions and / or reaches the maximum number of iterations, at which point the iteration process ends. Based on the updated state variables, the target Plück coordinates of the target rod are determined.

4. The method according to claim 3, wherein, The step of iteratively updating the initial state quantity using the Gauss-Newton method based on the target rod information included in each of the first images until the updated state quantity meets the preset conditions and / or reaches the maximum number of iterations, and then ending the iteration process, includes: During the iteration process, for the current iteration, based on the target rod information included in each of the first images, the odometry pose corresponding to each of the first images, and the first state quantity obtained in the previous iteration, the current reprojection residual and the current Jacobian matrix corresponding to the first state quantity are determined. The current reprojection residual includes the reprojection residual of the first state quantity on each of the first images, and the current Jacobian matrix includes the Jacobian matrix of the current reprojection residual with respect to the first state quantity. When the current iteration is the first iteration, the first state quantity is the initial state quantity. Based on the current reprojection residual and the current Jacobian matrix corresponding to the first state variable, determine the current step size corresponding to the current iteration; Based on the current step size and the first state quantity, determine the updated second state quantity; The iteration process ends when the reprojection residual corresponding to the second state variable meets the preset condition and / or the current iteration reaches the maximum number of iterations. Determining the target Plück coordinates of the target rod based on the updated state variables includes: Based on the second state variable, the target Plück coordinates of the target rod are determined.

5. The method according to claim 4, wherein, The target rod information includes the pixel coordinates of the two endpoints of the target rod in the first image; The step of determining the current reprojection residual corresponding to the first state quantity based on the target rod information included in each of the first images, the odometry pose corresponding to each of the first images, and the first state quantity obtained in the previous iteration includes: For each frame of the first image, based on the odometry pose corresponding to the first image and the first state quantity, a first intermediate quantity in the camera coordinate system corresponding to the first state quantity is determined; Based on the first intermediate quantity, determine the projection line of the first state quantity onto the first image; Based on the projection line and the pixel coordinates of the two ends of the target rod in the first image, the reprojection residual of the first state quantity in the first image is determined; The current reprojection residual is determined based on the reprojection residual of the first state quantity in each of the first images.

6. The method according to claim 5, wherein, Based on the target rod information included in each of the first images and the first state quantity obtained in the previous iteration, the current Jacobian matrix corresponding to the first state quantity is determined, including: Based on the pixel coordinates of the projection line and the two endpoints of the target rod in the first image, the first Jacobian of the reprojection residual with respect to the projection line is determined. Based on the projection matrix from the first intermediate quantity to the projected line, determine the second Jacobian of the projected line with respect to the first intermediate quantity; Based on the transformation matrix from the first state quantity to the camera coordinate system, determine the third Jacobian of the first intermediate quantity with respect to the first state quantity; Based on the transformation relationship between the first state variable and the orthogonal state variable, determine the fourth Jacobian of the first state variable with respect to the orthogonal state variable; Based on the first Jacobian, the second Jacobian, the third Jacobian, and the fourth Jacobian, the fifth Jacobian corresponding to the first state quantity in the first image is determined; The current Jacobian matrix is ​​determined based on the fifth Jacobian corresponding to the first state variable in each first image.

7. The method according to claim 4, wherein, Determining the current step size corresponding to the current iteration based on the current reprojection residual corresponding to the first state variable and the current Jacobian matrix includes: Based on the current Jacobian matrix, determine the Hessian matrix; Based on the Hessian matrix, the current Jacobian matrix, and the current reprojection residual, determine the normal equation regarding the iteration step size; The current step size is determined based on the normal equation.

8. The method according to claim 1, wherein, The step of determining the target rod-shaped object information included in each of the first images based on the first image sequence includes: For each frame of the first image in the first image sequence, a pre-trained rod detection model is used to determine the pixel region corresponding to the rod included in the first image; Based on the pixel regions corresponding to the rod-shaped objects included in each of the first images, a preset target tracking algorithm is used to track the target rod-shaped objects in the rod-shaped objects, and the target rod-shaped object information included in each of the first images is determined.

9. The method according to claim 1, wherein, Determining the position of the target rod-shaped object based on the target Plück coordinates includes: Based on the target rod-shaped object information included in each of the first images and the preset rules, a first target image and a second target image are determined from each of the first images; Based on the first camera optical center coordinates in the first camera coordinate system corresponding to the first target image, the second camera optical center coordinates in the first camera coordinate system corresponding to the second target image, and the first target endpoint and the second target endpoint in the first camera coordinate system corresponding to the first endpoint and the second target endpoint of the target rod on the first target image respectively, the first plane and the second plane in the first camera coordinate system are determined. Transform the first plane and the second plane to the world coordinate system to obtain the third plane and the fourth plane in the world coordinate system; Based on the third plane, the fourth plane, and the target Plück coordinates, determine the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system; Based on the three-dimensional endpoint coordinates of the two ends of the target rod in the world coordinate system, the three-dimensional coordinates of the target rod are determined; The three-dimensional coordinates are used as the position of the target rod.

10. A device for determining the position of a rod-shaped object, comprising: The first acquisition module is used to acquire a first image sequence and an odometer information sequence corresponding to the first image sequence. The first image sequence includes multiple frames of first images, and the odometer information sequence includes the odometer pose corresponding to each of the first images. The first processing module is used to determine the target rod-shaped object information included in each of the first images based on the first image sequence; The second processing module is used to determine the target Plück coordinates of the target rod based on the target rod information included in each of the first images and the odometer information sequence. The third processing module is used to determine the position of the target rod-shaped object based on the target Plück coordinates; The target Plück coordinates are obtained as follows: the target rod is represented by Plück coordinates as a state variable, and the state variable is iteratively updated based on the observation of the target rod in each of the first images in the first image sequence to obtain the target Plück coordinates; The position of the target rod is obtained as follows: based on the target rod information included in each of the first images and preset rules, a first target image and a second target image are determined from each of the first images; based on the first camera optical center corresponding to the first target image, the second camera optical center corresponding to the second target image, and the first target endpoint and the second target endpoint in the first camera coordinate system corresponding to the first endpoint and the second target endpoint of the target rod on the first target image, respectively, a third plane and a fourth plane in the world coordinate system are determined; based on the intersection of the third plane and the fourth plane with the spatial line represented by the target Plück coordinates, the position of the target rod is determined.

11. A computer-readable storage medium storing a computer program for performing the method for determining the position of the rod-shaped object according to any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining the position of the rod-shaped object according to any one of claims 1-9.

Images