A method of self-motion estimation and related apparatus

Abstract

This application provides a self-motion estimation method and related apparatus. The method includes: acquiring a first rotational velocity vector ω of the sensor's self-motion. s The normalized or scaled first translational velocity vector is based on the first rotational velocity vector ω. s The first translational velocity vector and the external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e In this embodiment, the carrier is the carrier on which the sensor is located. The external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system. By adopting the embodiments of this application, the motion speed of the carrier on which the sensor is located or the sensor can be determined, thereby improving the accuracy of the self-motion estimation of the carrier on which the sensor is located or the sensor, and thus improving the safety of assisted driving, autonomous driving or unmanned driving.

Description

Technical Field

[0001] This application relates to the field of sensor technology, and in particular to a self-motion estimation method and related apparatus. Background Technology

[0002] Advanced driver-assistance systems (ADAS) or autonomous driving systems (AD) typically employ various sensors, such as cameras, to perceive information about the surrounding environment, including moving and stationary targets. Moving targets include vehicles and pedestrians; stationary targets include obstacles, guardrails, light poles, and buildings. Different methods are used to analyze and process moving and stationary targets, such as classifying, identifying, and tracking moving targets, and classifying and identifying stationary targets to provide additional information for autonomous driving, such as obstacle avoidance and identifying drivable areas. Sensors can typically be mounted on various platforms, such as vehicles, ships, satellites, drones, and robots, and will move with the platform on which they are mounted. On the one hand, sensor motion makes it impossible to analyze moving and stationary targets independently; therefore, it is necessary to estimate sensor motion to separate moving and stationary targets. On the other hand, moving target tracking is usually model-based, and these models typically assume a coordinate system relative to the ground or geodetic coordinate system. However, sensor motion will cause these models to fail or tracking performance to degrade. In this case, compensation for the motion of the sensor or its carrier is required. Furthermore, for different real-world scenarios, it is also crucial to achieve the localization and tracking of sensor motion platforms through sensor motion estimation.

[0003] Therefore, determining the motion of the sensor and the carrier on which the sensor is located is a technical problem that those skilled in the art are currently working on. Summary of the Invention

[0004] This application discloses a self-motion estimation method and related apparatus, which can determine the velocity of the sensor and the carrier on which the sensor is located, thereby improving the accuracy of self-motion estimation of the sensor or the carrier on which the sensor is located.

[0005] The first aspect of this application discloses a self-motion estimation method, including:

[0006] Obtain the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling

[0007] According to the first rotational velocity vector ω s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e The carrier is the carrier on which the sensor is located, and the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system.

[0008] In the above method, based on the first rotational velocity vector ω of the sensor's self-motion s The first translational velocity vector after normalization or scaling The transformation relationship between the sensor coordinate system and the carrier coordinate system determines the carrier's second rotational velocity vector ω. e Second translational velocity vector t e Among them, the second translational velocity vector t e To obtain the complete translational velocity vector, while in existing technologies, the first translational velocity vector is typically obtained as a normalized or scaled version of the sensor's self-motion. Typically, information about one degree of freedom is lacking, but the embodiments of this application can accurately obtain the second rotational velocity vector ω of the carrier. e and the complete second translational velocity vector t e This means that missing information is recovered through external parameters, thereby more accurately estimating the speed of the carrier on which the sensor is located and the speed of the sensor itself. This improves the accuracy of the self-motion estimation of the sensor or the carrier on which the sensor is located, thereby improving the safety of assisted driving, autonomous driving or driverless driving.

[0009] Optionally, in one implementation, the transformation relationship includes: the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system.

[0010] Alternatively, in another implementation, the step of determining the first rotational velocity vector ω... s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e This includes: based on the rotation matrix R and the first rotational velocity vector ω s Determine the second rotational velocity vector ω of the carrier. e .

[0011] In the above method, the external parameters of the sensor coordinate system relative to the carrier coordinate system are used, namely the rotation matrix R, quaternions, Euler angles, and the first rotational velocity vector ω. s Determine the second rotational velocity vector ω of the carrier. eThis method can effectively utilize the external parameters of the sensor relative to the carrier to obtain the rotational angular velocity of the carrier where the sensor is located, thereby effectively improving the accuracy of the rotational velocity estimation of the carrier where the sensor is located.

[0012] Alternatively, in another implementation, the step of determining the first rotational velocity vector ω... s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e This includes: based on the second rotational velocity vector ω e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0013] Alternatively, in another implementation, the step of determining the second translational velocity vector t of the carrier based on the instantaneous velocity component and the normalized instantaneous velocity vector is... e This includes: determining the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and determining the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

[0014] Alternatively, in another implementation, the second translational velocity vector t of the carrier e , s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

[0015] Optionally, in another implementation, the method further includes: based on the amplitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector... Determine the third translational velocity vector t of the sensor's self-motion. s .

[0016] In the above method, the amplitude s of the translational velocity vector in the sensor coordinate system and the first translational velocity vector are used. Determine the third translational velocity vector t of the sensor's self-motion. s Among them, the first translational velocity vector For normalization or scaling, and the third translational velocity vector t s This is the complete translational velocity vector. However, existing technologies can only obtain the normalized or scaled first translational velocity vector. Compared with the prior art, the embodiments of this application provide a complete third translational velocity vector t for determining the self-motion of a sensor in the absence of scale information. s The method.

[0017] Optionally, in another implementation, the sensor is a vision sensor, and the method further includes: acquiring the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the vision sensor; and based on the flow vector of the stationary obstacle and the third translational velocity vector t... s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0018] In the above method, the flow vector of the stationary obstacle and the third translational velocity vector t are used. s and the first rotational velocity vector ω s Determining the depth z of a stationary obstacle is a challenge. However, existing technologies suffer from scale issues, where depth information is coupled with various components of translational velocity, making accurate depth estimation difficult. Compared to existing technologies, embodiments of this application can obtain a more accurate depth z by utilizing flow vectors.

[0019] A second aspect of this application discloses a self-motion estimation device, comprising:

[0020] The first acquisition unit is used to acquire the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling

[0021] The first determining unit is configured to determine the first rotational velocity vector ω. s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e The carrier is the carrier on which the sensor is located, and the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system.

[0022] In the above device, based on the first rotational velocity vector ω of the sensor's self-motion... s The first translational velocity vector after normalization or scaling The transformation relationship between the sensor coordinate system and the carrier coordinate system determines the carrier's second rotational velocity vector ω. e Second translational velocity vector t e Among them, the second translational velocity vector t e To obtain the complete translational velocity vector, while in existing technologies, the first translational velocity vector is typically obtained as a normalized or scaled version of the sensor's self-motion. Typically, information about one degree of freedom is lacking; therefore, the embodiments of this application are able to accurately obtain the second rotational velocity vector ω of the carrier. e and the complete second translational velocity vector t e This means that by recovering the missing information through the external parameters of the sensor, the speed of the carrier on which the sensor is located and the speed of the sensor can be estimated more accurately. This improves the accuracy of the self-motion estimation of the sensor or the carrier on which the sensor is located, thereby improving the safety of assisted driving, autonomous driving or driverless driving.

[0023] Optionally, in one implementation, the transformation relationship includes: the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system.

[0024] Optionally, in another implementation, the first determining unit is configured to determine the rotation matrix R and the first rotational velocity vector ω. s Determine the second rotational velocity vector ω of the carrier. e .

[0025] In the aforementioned device, based on the external parameters of the sensor coordinate system relative to the carrier coordinate system, namely the rotation matrix R, quaternions, Euler angles, and the first rotational velocity vector ω... s Determine the second rotational velocity vector ω of the carrier. e This method can effectively utilize the external parameters of the sensor relative to the carrier to obtain the rotational angular velocity of the carrier where the sensor is located, thereby effectively improving the accuracy of the rotational velocity estimation of the carrier where the sensor is located.

[0026] Optionally, in another implementation, the first determining unit is configured to determine the second rotational velocity vector ω. e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0027] Optionally, in another implementation, the first determining unit is configured to determine the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and to determine the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

[0028] Optionally, in another implementation, the second translational velocity vector t of the carrier e , s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

[0029] Optionally, in another implementation, the device further includes: a second determining unit, configured to determine the magnitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector. Determine the third translational velocity vector t of the sensor's self-motion. s .

[0030] In the aforementioned device, the amplitude s of the translational velocity vector in the sensor coordinate system and the first translational velocity vector... Determine the third translational velocity vector t of the sensor's self-motion. s Among them, the first translational velocity vector For normalization or scaling, and the third translational velocity vector t s This is the complete translational velocity vector. However, existing technologies can only obtain the normalized or scaled first translational velocity vector. Compared with the prior art, the embodiments of this application provide a complete third translational velocity vector t for determining the self-motion of a sensor in the absence of scale information. s The method.

[0031] Optionally, in another implementation, the sensor is a vision sensor, and the device further includes: a second acquisition unit for acquiring the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the vision sensor; and a third determination unit for determining the flow vector of the stationary obstacle and the third translational velocity vector t. s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0032] In the aforementioned device, the flow vector passing through the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ωs Determining the depth z of a stationary obstacle is a challenge. However, existing technologies suffer from scale issues, where depth information is coupled with various components of translational velocity, making accurate depth estimation difficult. Compared to existing technologies, embodiments of this application can obtain a more accurate depth z by utilizing optical flow vectors.

[0033] A third aspect of this application discloses a self-motion estimation device, which includes at least one processor and a communication interface, and optionally, a memory. The memory stores a computer program, and the at least one processor invokes the computer program to perform the following operations:

[0034] Obtain the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling According to the first rotational velocity vector ω s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e The carrier is the carrier on which the sensor is located, and the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system.

[0035] In the above device, based on the first rotational velocity vector ω of the sensor's self-motion... s The first translational velocity vector after normalization or scaling The transformation relationship between the sensor coordinate system and the carrier coordinate system determines the carrier's second rotational velocity vector ω. e Second translational velocity vector t e Among them, the second translational velocity vector t e To obtain the complete translational velocity vector, while in existing technologies, the first translational velocity vector is typically obtained as a normalized or scaled version of the sensor's self-motion. Typically, information about one degree of freedom is lacking, but the embodiments of this application can accurately obtain the second rotational velocity vector ω of the carrier. e and the complete second translational velocity vector t e This means that missing information is recovered by using the external parameters of the sensor, thereby more accurately estimating the speed of the carrier on which the sensor is located and the speed of the sensor itself. This improves the accuracy of the self-motion estimation of the sensor or the carrier on which the sensor is located, and thus improves the safety of autonomous driving.

[0036] Optionally, in one implementation, the transformation relationship includes: the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system.

[0037] Alternatively, in another implementation, the at least one processor is configured to, based on the rotation matrix R and the first rotational velocity vector ω s Determine the second rotational velocity vector ω of the carrier. e .

[0038] In the aforementioned device, based on the external parameters of the sensor coordinate system relative to the carrier coordinate system, namely the rotation matrix R, quaternions, Euler angles, and the first rotational velocity vector ω... s Determine the second rotational velocity vector ω of the carrier. e This method can effectively utilize the external parameters of the sensor relative to the carrier to obtain the rotational angular velocity of the carrier where the sensor is located, thereby effectively improving the accuracy of the rotational velocity estimation of the carrier where the sensor is located.

[0039] Alternatively, in another implementation, the at least one processor is configured to, based on the second rotational velocity vector ω e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0040] Optionally, in another implementation, the at least one processor is configured to determine the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and to determine the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

[0041] Alternatively, in another implementation, the second translational velocity vector t of the carrier e , s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

[0042] Optionally, in another implementation, the at least one processor is further configured to, based on the magnitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector... Determine the third translational velocity vector t of the sensor's self-motion. s .

[0043] Optionally, in the above-described device, the amplitude s of the translational velocity vector in the sensor coordinate system and the first translational velocity vector are... Determine the third translational velocity vector t of the sensor's self-motion. s Among them, the first translational velocity vector For normalization or scaling, and the third translational velocity vector t s This is the complete translational velocity vector. However, existing technologies can only obtain the normalized or scaled first translational velocity vector. Compared with the prior art, the embodiments of this application provide a complete third translational velocity vector t for determining the self-motion of a sensor in the absence of scale information. s The method.

[0044] Optionally, in another implementation, the sensor is a vision sensor, and the at least one processor is further configured to acquire the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the vision sensor; based on the flow vector of the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0045] In the aforementioned device, the flow vector passing through the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ω s Determining the depth z of a stationary obstacle is a challenge. However, existing technologies suffer from scale issues, where depth information is coupled with various components of translational velocity, making accurate depth estimation difficult. Compared to existing technologies, embodiments of this application can obtain a more accurate depth z by utilizing optical flow vectors.

[0046] The fourth aspect of this application discloses a computer product that, when the computer program product is run on a processor, implements the method described in any aspect or any optional scheme of any aspect.

[0047] The fifth aspect of this application discloses a chip system, which includes at least one processor and a communication interface. Optionally, the chip system also includes a memory. The at least one processor is used to call a computer program stored in at least one memory so that the device in which the chip system is located implements the method described in any aspect or any optional scheme.

[0048] The sixth aspect of this application discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any aspect or any optional scheme of any aspect.

[0049] A seventh aspect of this application discloses a terminal, which includes at least one processor and a communication interface, and optionally, a memory. The at least one processor is used to call a computer program stored in at least one memory so that the terminal implements the method described in any aspect or any optional scheme.

[0050] The eighth aspect of this application discloses a sensor, which includes at least one processor and a communication interface. Optionally, the sensor also includes a memory. The at least one processor is used to call a computer program stored in at least one memory so that the device in which the sensor is located implements the method described in any aspect or any optional scheme. Attached Figure Description

[0051] The accompanying drawings used in the embodiments of this application are described below.

[0052] Figure 1 This is a schematic diagram of the structure of a self-motion estimation system provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of an inertial measurement unit provided in an embodiment of this application;

[0054] Figure 3 This is a flowchart of a self-motion estimation method provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram illustrating a representation of a sensor coordinate system provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram illustrating a representation of a carrier coordinate system provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a self-motion estimation device provided in an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of another self-motion estimation device provided in the embodiments of this application. Detailed Implementation

[0059] The embodiments of this application are described below with reference to the accompanying drawings.

[0060] Please see Figure 1 , Figure 1This is a schematic diagram of a self-motion estimation system provided in an embodiment of this application. The system includes a sensor 1001, a motion measurement module 1002, and a data processing module 1003. The sensor 1001 can be a visual sensor, such as an infrared thermal imaging sensor, a camera, or a webcam. The sensor 1001 provides visual measurement data, such as images or videos. The motion measurement module 1002 acquires motion measurement data based on the sensor data, such as the first rotational velocity vector ω of the sensor's self-motion. s and the first translational velocity vector after normalization or scaling. The data processing module 1003 is used to process the measurement data provided by the sensor 1002. Under normal circumstances, the motion measurement module 1002 and the data processing module 1003 can be in the same processor.

[0061] In one example, sensor 1001, motion measurement module 1002, and data processing module 1003 can be fully or partially integrated via wired or wireless means, for example, sensor 1001, motion measurement module 1002, and data processing module 1003 can be deployed on a single processor system; in this case, the self-motion estimation system can be located on the same vehicle, airborne, spaceborne, or intelligent agent. In another example, sensor 1001 is located on a vehicle, airborne, spaceborne, or intelligent agent, while motion measurement module 1002 and / or data processing module 1003 are located in the cloud. Accordingly, sensor 1001 sends the provided video data or parameterized data, or the data provided by motion measurement module 1002, to data processing module 1003 in the cloud, and data processing module 1003 sends the processed results back to the vehicle, airborne, spaceborne, or intelligent agent. The vehicle can be a car, motorcycle, or bicycle; the airborne can be a drone, helicopter, or jet aircraft; the spaceborne can be a satellite; and the intelligent agent can be a robot.

[0062] Currently, there are various methods for obtaining motion speed. In one method, the vehicle is typically equipped with an inertial measurement unit (IMU), such as... Figure 2 As shown, Figure 2An Inertial Measurement Unit (IMU) is a device that measures an object's three-axis attitude angles (or angular velocities) and acceleration. Typically, an IMU contains three single-axis gyroscopes and three single-axis accelerometers. The gyroscopes detect the angular velocity signal of the vehicle relative to the navigation coordinate system, while the accelerometers detect the acceleration signal of the object along three independent axes within the vehicle's coordinate system. By measuring the object's angular velocity and acceleration in three-dimensional space, the object's motion velocity and attitude can be calculated. However, in the above method, the object's motion velocity is usually based on the accumulation of acceleration. But the measurement error of acceleration measured by accelerometers accumulates over time, resulting in an error accumulation problem. This necessitates additional calibration using other sensors. Moreover, the accuracy of IMUs typically used in vehicles is too low, and using high-precision IMUs would be prohibitively expensive.

[0063] In another approach, vehicles are typically equipped with millimeter-wave radar sensors, which usually provide measurements such as range, azimuth, and radial velocity. Based on the azimuth and radial velocity components of a stationary target, the sensor's instantaneous velocity relative to the ground can be obtained using the least squares method or other means. However, in the aforementioned methods, only two velocity components relative to the radar sensor coordinate system—lateral and radial velocity—are typically obtained; a third velocity component cannot be acquired. Furthermore, while the radial velocity error is relatively low, the lateral velocity error is relatively high, making it difficult to meet the system's performance requirements. Moreover, this method can only obtain one rotational angular velocity component, namely the yaw rate, and cannot obtain the pitch rate and roll rate.

[0064] In another approach, a camera or video camera is installed in the vehicle to provide continuous images. Based on the two or more frames of images, the sensor's translational and rotational velocity vectors relative to the ground (normalized or scaled) can be obtained using optical flow, feature point methods, or methods that directly optimize the objective function of brightness. However, this method suffers from scale issues, and the various components of depth and translational velocity are coupled together, often making it impossible to obtain more accurate depth information. Consequently, it is impossible to obtain precise translational velocity, or only normalized or scaled translational velocity can be obtained.

[0065] Please see Figure 3 , Figure 3This application provides a self-motion estimation method. The execution entity of this method can be a sensor system, a fusion perception system, or a planning / control system integrating the above systems, such as an assisted driving or autonomous driving system. Alternatively, the execution entity can also be software or hardware (such as a data processing device connected wirelessly or wiredly to or integrated with the corresponding sensors). The following different execution steps can be implemented centrally, or they can be implemented in a distributed manner. The method includes, but is not limited to, the following steps:

[0066] Step S301: Obtain the first rotational velocity vector ω of the sensor's self-motion from the sensor. s The first translational velocity vector after normalization or scaling

[0067] Specifically, the first translational velocity vector It can also be called the first translational velocity vector. The sensor can be a visual sensor such as a camera, webcam, infrared sensor, or other imaging sensor, etc., and the embodiments of this application are not limited thereto.

[0068] Specifically, the first rotational velocity vector ω of the sensor's self-motion is obtained. s and the first translational velocity vector The data can be obtained via wired or wireless means. For example, it can be read directly within the same processor module; within the same hardware system, it can be obtained via a bus such as the Peripheral Component Interconnect (PCI) bus, or via a network such as various types of controller area networks (CAN) in a vehicle; or via wireless means such as cloud communication. Of course, it can also be obtained in other forms, which are not limited in the embodiments of this application.

[0069] Specifically, the sensor's first rotational velocity vector ω s and the first translational velocity vector The optical or geometric properties of the data can be determined based on the feature points, lines, planes, or regions in the data acquired by the sensor, such as the 8-point method, 5-point method, homography, or optical flow method. This application does not limit the specific methods.

[0070] For example, the first rotational velocity vector ω s It can be represented as ω s =[ω s，x ω s，y ω s，z ] T, where ω s，x ω s，y and ω s，z The components of the sensor's first rotational velocity along the three coordinate axes; the normalized or scaled first translational velocity vector. It can be represented as Specifically, It can be the normalized vector value of the sensor's first translational velocity vector t′ or the scaled vector value; for example, the normalized or scaled first translational velocity vector. The scaling or normalization of the sensor's first translational velocity vector t′ is determined by the scale.

[0071]

[0072] Where t′ is the first translational velocity vector of the sensor's self-motion, it can be expressed as:

[0073] t′=[t′ x t′ y t′ z ] T ,

[0074] The normalized value s′ or the scaling value s′ can be the magnitude or norm of the vector t′, i.e., s′=||t′||.

[0075] or,

[0076]

[0077] The normalized value s′ or the scaling value s′ can also be a component of the vector t′, for example, it can be the third component t′ of the vector t′. z ,Right now:

[0078] s′=t′ z ,

[0079] The normalized value s′ or the scale value s′ can also be other unknown scalars, such as depth, which is not limited here.

[0080] Optionally, as one implementation, the sensor or the carrier on which the sensor resides moves within a plane, such as the ground or a planar track. The first rotational velocity vector ω s It can be represented as ω s =[0 0ω s，z ] T The component ω of the sensor's first rotational velocity in the X-axis direction is... s，x =0, the component ω of the sensor's first rotational velocity in the Y-axis direction. s，y=0; the first translational velocity vector after normalization or scaling. It can be represented as At this time, the first rotational velocity vector ω s It can be simplified to using ω s，z express, It can be simplified to use express.

[0081] Step S302: Based on the first rotational velocity vector ω s The first translational velocity vector after normalization or scaling The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e and / or the second translational velocity vector t e .

[0082] Specifically, the second translational velocity vector t of the carrier e For a complete translational velocity vector, the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system. These external parameters may include the translation parameters of the sensor coordinate system relative to the carrier coordinate system and / or the rotation parameters of the sensor coordinate system relative to the carrier coordinate system; or, they may include the translation parameters of the carrier coordinate system relative to the sensor coordinate system and / or the rotation parameters of the carrier coordinate system relative to the sensor coordinate system.

[0083] Specifically, the translation of the sensor coordinate system relative to the carrier coordinate system can be a translation vector of the origin of the sensor coordinate system relative to the origin of the carrier coordinate system; or, the position vector of the origin of the sensor coordinate system within the origin of the carrier coordinate system, which is not limited here. The rotation of the sensor coordinate system relative to the carrier coordinate system can be represented by a rotation matrix, quaternion, or Euler angles of the sensor coordinate system relative to the carrier coordinate system, which is not limited here.

[0084] Specifically, the translation of the carrier coordinate system relative to the sensor coordinate system can be a translation vector of the origin of the carrier coordinate system relative to the origin of the sensor coordinate system; or, the position vector of the origin of the carrier coordinate system within the origin of the sensor coordinate system, which is not limited here. The rotation of the carrier coordinate system relative to the sensor coordinate system can be represented by a rotation matrix, quaternion, or Euler angles of the rotation of the carrier coordinate system relative to the sensor coordinate system, which is not limited here.

[0085] Specifically, the external parameters of the sensor coordinate system relative to the carrier coordinate system are related to the sensor's mounting position on the carrier. For example, when the sensor is mounted at the center of the bumper of the vehicle body (carrier), the sensor coordinate system is as follows: Figure 4 As shown, the X-axis is usually forward, and a right-handed coordinate system is used. Figure 4This is a representation of a sensor coordinate system, such as O1XYZ. The carrier coordinate system is fixed to the carrier, and its origin is the center of the carrier. For example, the origin of a vehicle coordinate system is located at the center of the rear axle. Figure 5 As shown, the X-axis is usually forward, and a right-handed coordinate system is used. Figure 5 This is a representation of a carrier coordinate system, such as using O2X. b Y b Z b It indicates that O2X b O2Y b Moving forward along the longitudinal axis of the carrier, O2Z b Upward along the vertical axis of the carrier.

[0086] Optionally, as one implementation, the transformation relationship may include the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system. The target position vector measured by the sensor is p. b The position vector in the carrier coordinate system is p e , can be represented as:

[0087] p e =Rp b +r.

[0088] Alternatively, as one implementation, the sensor or the carrier on which the sensor is located moves in a plane, and the aforementioned coordinate rotation can be represented by rotation about the z-axis.

[0089] In one implementation, based on the rotation matrix R and the first rotational velocity vector ω s Determine the second rotational velocity vector ω of the carrier e .

[0090] Specifically, the second rotational velocity vector ω of the carrier e Let R be the rotation matrix and ω be the first rotational velocity vector. s The product of . For example, if the rotation matrix is ​​R, and the first rotational velocity vector is ω s The second rotational velocity vector of the carrier is determined as follows:

[0091] ω e =Rω s .

[0092] Since the rotation of the sensor coordinate system relative to the carrier coordinate system can be represented by a rotation matrix, quaternion, or Euler angles, the second rotational velocity vector of the carrier can also be determined based on the quaternion or Euler angles. This will not be elaborated further here.

[0093] In one example, the sensor coordinate system and the carrier coordinate system have the same axis directions. In this case, the second rotational velocity vector is equal to the first rotational velocity vector, ω. e =ω s At this point, R is the identity matrix I.

[0094] In one implementation, based on the second rotational velocity vector ω e and the first translational velocity vector after normalization or scaling. In addition to the external parameters of the sensor, the second translational velocity vector t of the carrier is determined. e .

[0095] Specifically, the external parameters of the sensor may include the translation vector r of the sensor coordinate system relative to the carrier coordinate system and / or rotation parameters, such as rotation matrices, quaternions, or Euler angles.

[0096] The sensor's external parameters include the translation vector r, which, in one implementation, is based on the second rotational velocity vector ω. e The instantaneous velocity components caused by the carrier rotation are determined by the translation vector r; based on the rotation matrix R and the normalized or scaled first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0097] Specifically, according to the second rotational velocity vector ω e The translation vector r determines the instantaneous velocity component t caused by the rotation of the carrier. R It can refer to the instantaneous velocity component t caused by the rotation of the carrier. R The second rotational velocity vector ω e The cross product of translation vector r; t R =ω e ×r.

[0098] Where × represents the cross product of vectors. Specifically, it can be:

[0099]

[0100] Where, ω e =[ω e，x ω e，y ω e，z ] T .

[0101] Or the instantaneous velocity component t caused by the rotation of the carrier. R Let r be the translation vector and ω be the second rotational velocity vector. e The negative vector of the cross product,

[0102] t R = -r×ω e ,

[0103] Where × represents the cross product of vectors. Specifically, it can be...

[0104] Where, r = [r x r y r z ] T Of course, other methods can also be used, and the embodiments in this application are not limited thereto.

[0105] Among them, the instantaneous velocity component t caused by the rotation of the aforementioned carrier R It can be a three-dimensional vector, such as t R =[t R，x t R，y t R，z ] T Or t R It can also be a two-dimensional vector.

[0106] Optionally, as one implementation, the sensor or the carrier on which the sensor is located moves within a plane, such as the ground or a planar track. In this case, the first rotational velocity vector ω... s It can be simplified to using ω s，z express, It can be simplified to use This means that the translation vector r = [r x r y r z ] T It can be simplified to using r x r y It means that r z =0. Instantaneous velocity component t R It can be simplified to:

[0107] t R，x =-ω e，z ·r y ,

[0108] t R，y =ω e，z ·r x ,

[0109] t R，z =0.

[0110] t R It can be simplified to a two-dimensional vector, including t R，x , t R，y Two components.

[0111] In one example, the first translational velocity vector can be determined based on rotation parameters and normalized or scaled. Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system.

[0112] Specifically, the rotation parameters of the sensor relative to the carrier, such as rotation matrices, quaternions, or Euler angles, are important parameters.

[0113] Taking the rotation matrix R as an example, based on the rotation matrix R and the first translational velocity vector after normalization or scaling... Determining the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system can refer to the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system. The rotation matrix R and the normalized or scaled first translational velocity vector are given. The product of, i.e.

[0114] In one implementation, R is the identity matrix, representing the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system and the normalized or scaled first translational velocity vector. equal,

[0115] It can be a three-dimensional vector. or, It can also be a two-dimensional vector, including Two components.

[0116] In one example, the second translational velocity vector t of the carrier is determined based on the instantaneous velocity components and the normalized instantaneous velocity vector. e It can refer to the second translational velocity vector t of the carrier. e The instantaneous velocity vector is the difference between the instantaneous velocity vector and the instantaneous velocity component, and the instantaneous velocity vector is determined based on the normalized instantaneous velocity vector and the instantaneous velocity component.

[0117] Specifically, the second translational velocity vector t of the carrier e for:

[0118]

[0119] Among them, t R，y For t R The second component, that is, the component along the y-axis, for The second component, that is, the component along the y-axis.

[0120] Specifically, the instantaneous velocity vector is determined based on the normalized instantaneous velocity vector and the instantaneous velocity components, including...

[0121] The instantaneous velocity vector is determined based on the normalized instantaneous velocity vector and the amplitude of the translational velocity vector in the sensor coordinate system, wherein the amplitude of the translational velocity vector in the sensor coordinate system is based on the instantaneous velocity component t. R The second component and the second component of the normalized instantaneous velocity vector are determined, and the amplitude of the translational velocity vector of the sensor coordinate system can be:

[0122]

[0123] In yet another example, the second translational velocity vector t of the carrier is determined based on the instantaneous velocity components and the normalized instantaneous velocity vector. e This includes: determining the amplitude of the translational velocity vector in the sensor coordinate system based on the instantaneous velocity components and the normalized instantaneous velocity vector; and determining the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector in the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity components. e .

[0124] Specifically, the amplitude of the translational velocity vector in the sensor coordinate system is determined based on the instantaneous velocity components and the normalized instantaneous velocity vector, including:

[0125]

[0126] Where s represents the magnitude of the translational velocity vector in the sensor coordinate system.

[0127] Specifically, the second translational velocity vector t of the carrier is determined based on the amplitude of the translational velocity vector in the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity components. e ,include:

[0128]

[0129] Where s is the magnitude of the translational velocity vector in the sensor coordinate system. For the normalized instantaneous velocity vector, t R This represents the instantaneous velocity component. Where t... e，y The velocity is 0 because, during the vehicle's design, if the vehicle rotates, the velocity at the center of the rear axle, i.e., along the y-axis of the vehicle's coordinate system, must be 0, i.e., t. e，y If t is 0, e，y If the value is not 0, the vehicle will skid; therefore, t e，y It is 0.

[0130] t e It can be a three-dimensional vector, t e =[t e，x t e，y t e，z ] T Or te It can also be a two-dimensional vector, including t e，x , t e，y Two velocity components.

[0131] Optionally, embodiments of this application may further include step S303:

[0132] Step S303: Based on the amplitude s of the translational velocity vector in the sensor coordinate system and the normalized or scaled first translational velocity vector... Determine the third translational velocity vector t of the sensor's self-motion. s .

[0133] Specifically, the calculation method for the amplitude s of the translational velocity vector of the sensor coordinate system is as described in step S302, and will not be repeated here. The third translational velocity vector t of the sensor's self-motion... s This is the complete translational velocity vector. It is determined by the amplitude *s* of the translational velocity vector in the sensor coordinate system and the first translational velocity vector after normalization or scaling. Determine the third translational velocity vector t of the sensor's self-motion. s ,include:

[0134]

[0135] It should be noted that the third translational velocity vector of the sensor's self-motion is the sensor's instantaneous velocity vector; or the third translational velocity vector of the sensor's self-motion is the instantaneous velocity vector of the origin of the sensor's coordinate system.

[0136] Furthermore, the third translational velocity vector of the sensor's self-motion can be transformed into the sensor coordinate system and expressed as t′. s ,like,

[0137] t′ s =R -1 t s ,

[0138] Optionally, embodiments of this application may further include step S304:

[0139] Step S304: Determine the depth z of the stationary obstacle.

[0140] Specifically, in step S304, the sensor is a vision sensor.

[0141] Step S304 includes steps S3041 and S3042. Specifically:

[0142] Step S3041: Obtain the flow vector of the stationary obstacle.

[0143] Specifically, the flow vector includes the motion vector of the projection of a stationary obstacle onto the image plane of the vision sensor. There are various methods for obtaining the flow vector of a stationary obstacle; the flow vector can be obtained through optical flow, for example, by algorithms such as the Lukas-Kanade optical flow algorithm or the Horn-Schunck optical flow algorithm. This application does not limit the specific methods described.

[0144] For ease of understanding, this application provides a method for obtaining the flow vector of a stationary obstacle using the Lukas-Kanade optical flow method, as follows:

[0145] Position of stationary obstacles on the image (p x p y The corresponding flow vector can be represented by the position (p) x p y The optical flow (u, v) corresponding to the given condition can be calculated, specifically, by using the following relationship:

[0146]

[0147] Where G is the gradient matrix and b is the mismatch vector, specifically,

[0148]

[0149]

[0150] Among them, I x (x, y) and I y (x, y) represents the gradient of the brightness of image point (x, y) in the x and y directions at that point; I t The change in image brightness over time, [p x -w x p x +w x ] and [p y -w y p y +w y ] is a point (p) x p y The corresponding window has a size of (2w) x +1)(2w y +1), w x and w y For pre-defined parameters, such as w x =w y =1, 2, 3...

[0151] It should be noted that even for the Lukas-Kanade optical flow algorithm, there are multiple implementations, such as pyramid-structure-based hierarchical estimation, which will not be elaborated here.

[0152] Step S3042: Based on the flow vector of the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0153] Specifically, the third translational velocity vector t of the sensor's self-motion s It can be transformed into the sensor coordinate system and represented as t′ s ω s Let be the first rotational velocity vector of the sensor's self-motion. The flow vector of a stationary obstacle can be represented as (u, v).

[0154] Based on the flow vector of the stationary obstacle and the third translational velocity vector t′ s and the first rotational velocity vector ω s The depth z of the stationary obstacle can be determined based on the following relationship:

[0155]

[0156]

[0157] Where, s1 = [-f 0 x] T s2 = [0 - fy] T , × represents the cross product of vectors, f is the camera focal length, and x and y are the pixel positions in the image plane, x∈[p x -w x p x +w x ], y∈[p y -w y p y +w y ]; where w x and w y w is a non-negative integer. x =0, 1, 2, 3, 4...; w y =0, 1, 2, 3, 4...

[0158] Specifically, the depth z of a stationary obstacle can be obtained according to the following relationship:

[0159]

[0160] Where w1(x, y) and w2(x, y) are weighting coefficients, which can be determined based on the parameters u, v and t′ mentioned above. s and ω s The accuracy or error of the estimation is determined.

[0161] Specifically, as an implementation,

[0162]

[0163] or

[0164]

[0165] or

[0166]

[0167] or

[0168]

[0169] Optionally, embodiments of this application may further include step S305.

[0170] Step S305: Based on the second rotational velocity vector ω of the carrier e Second translational velocity vector t e And the rotational velocity vector ω of the carrier obtained through other sensors. e2 Translational velocity vector t e2 Determine the rotational velocity vector of the carrier Translational velocity vector

[0171] Specifically, the second rotational velocity vector ω of the carrier e Second translational velocity vector t e The rotational velocity vector ω of the carrier can be obtained based on data provided by a vision sensor. e2 Translational velocity vector t e2 It can be obtained based on data provided by radar sensors or other sensors, in which one or more sets of rotational velocity vectors and translational velocity vectors of the carrier can be obtained through other sensors.

[0172] For example, for ease of understanding, in the embodiments of this application, in addition to obtaining the second rotational velocity vector ω of the carrier through a visual sensor... e Second translational velocity vector t e In addition, the rotational velocity vector ω of the carrier is obtained through radar sensors. e2 Translational velocity vector t e2 Therefore, the weighted rotational velocity vector of the carrier... It can be obtained from the following formula:

[0173]

[0174] Where the weighting coefficients w1 and w2 can be... Or based on ω e and ω e2 The error covariance or standard deviation is determined, for example, by the inverse matrix or reciprocal of the covariance or standard deviation.

[0175] Weighted translational velocity vector It can be obtained from the following formula:

[0176]

[0177] Where the weighting coefficients w1 and w2 can be... Or based on ω e and ω e2 The error covariance or standard deviation is determined, for example, by the inverse matrix or reciprocal of the covariance or standard deviation.

[0178] In addition to the weighting methods described above, other weighting methods are also included, but this application does not limit the specific weighting methods used in its embodiments.

[0179] exist Figure 3 In the described method, based on the first rotational velocity vector ω of the sensor's self-motion s The first translational velocity vector after normalization or scaling The transformation relationship between the sensor coordinate system and the carrier coordinate system determines the carrier's second rotational velocity vector ω. e Second translational velocity vector t e Among them, the second translational velocity vector t e To obtain the complete translational velocity vector, while in existing technologies, the first translational velocity vector is typically obtained as a normalized or scaled version of the sensor's self-motion. Typically, information about one degree of freedom is lacking, but the embodiments of this application can accurately obtain the second rotational velocity vector ω of the carrier. e and the complete second translational velocity vector t e This means that by recovering the missing information through the external parameters of the sensor, the speed of the carrier on which the sensor is located and the speed of the sensor can be estimated more accurately. This improves the accuracy of the self-motion estimation of the sensor or the carrier on which the sensor is located, thereby improving the safety of assisted driving, autonomous driving or driverless driving.

[0180] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0181] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a self-motion estimation device provided in an embodiment of this application. The self-motion estimation device may include a first acquisition unit 601 and a first determination unit 602, wherein the detailed description of each unit is as follows.

[0182] The first acquisition unit 601 is used to acquire the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling

[0183] The first determining unit 602 is configured to determine the first rotational velocity vector ω based on the first determining unit 602. s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e The carrier is the carrier on which the sensor is located, and the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system.

[0184] In one implementation, the transformation relationship includes: the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system.

[0185] In another implementation, the first determining unit 602 is configured to determine the rotation matrix R and the first rotational velocity vector ω. s Determine the second rotational velocity vector ω of the carrier. e .

[0186] In another implementation, the first determining unit 602 is configured to determine the second rotational velocity vector ω. e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0187] Optionally, in another implementation, the first determining unit 602 is configured to determine the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and to determine the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

[0188] Alternatively, in another implementation, the second translational velocity vector t of the carrier e ,

[0189]

[0190] s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

[0191] Optionally, in another implementation, the device further includes: a second determining unit, configured to determine the magnitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector based on... Determine the third translational velocity vector t of the sensor's self-motion. s .

[0192] Optionally, in another implementation, the sensor is a vision sensor, and the device further includes: a second acquisition unit for acquiring the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the vision sensor; and a third determination unit for determining the flow vector of the stationary obstacle and the third translational velocity vector t. s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0193] It should be noted that the implementation and beneficial effects of each unit can be found by referring to [the relevant documentation / reference]. Figure 3 The corresponding description of the method embodiments shown.

[0194] Please see Figure 7 , Figure 7 This application provides a self-motion estimation device 700, which includes at least one processor 701 and a communication interface 702, and optionally, a memory 703. The memory 703 is used to store computer programs, and the at least one processor 701, the memory 703, and the communication interface 702 are interconnected via a bus 704.

[0195] The memory 703 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related computer programs and data. The communication interface 702 is used for receiving and sending data.

[0196] Processor 701 can be one or more central processing units (CPUs). When processor 701 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0197] The processor 701 in the device 700 reads the computer program stored in the memory 703 and performs the following operations:

[0198] Obtain the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling

[0199] According to the first rotational velocity vector ω s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e The carrier is the carrier on which the sensor is located, and the external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system.

[0200] Optionally, in one implementation, the transformation relationship includes: the rotation matrix R and translation vector r of the sensor coordinate system relative to the carrier coordinate system.

[0201] Alternatively, in another implementation, the processor is configured to, based on the rotation matrix R and the first rotational velocity vector ω s Determine the second rotational velocity vector ω of the carrier. e .

[0202] Alternatively, in another implementation, the processor is configured to, based on the second rotational velocity vector ω e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e .

[0203] Optionally, in another implementation, the processor is configured to determine the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and to determine the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

[0204] Alternatively, in another implementation, the second translational velocity vector t of the carrier e ,

[0205]

[0206] s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

[0207] In another implementation, the processor is further configured to, based on the magnitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector... Determine the third translational velocity vector t of the sensor's self-motion. s .

[0208] In another implementation, the sensor is a vision sensor, and the processor is further configured to acquire the flow vector of a stationary obstacle, wherein the flow amount includes the motion vector of the stationary obstacle on the image plane of the vision sensor; based on the flow vector of the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

[0209] It should be noted that the implementation and beneficial effects of each operation can be found by referring to [the relevant documentation / reference]. Figure 3 The corresponding description of the method embodiments shown.

[0210] This application embodiment also provides a chip system, the chip system including at least one processor and a communication interface. Optionally, the chip system further includes a memory, the at least one processor being used to call a computer program stored in the at least one memory, so that... Figure 3 The method and flow shown are thus implemented.

[0211] This application also provides a computer-readable storage medium storing a computer program, which, when executed... Figure 3 The method and flow shown are thus implemented.

[0212] This application also provides a computer program product, which, when run, provides a method for... Figure 3 The method and flow shown are thus implemented.

[0213] This application embodiment also provides a terminal, which includes at least one processor and a communication interface, and optionally, a memory. The at least one processor is used to call a computer program stored in at least one memory, so that... Figure 3 The illustrated method flow is thus implemented. Optionally, the terminal can be a vehicle, ship, satellite, drone, robot, smart home device, or smart manufacturing equipment, etc.

[0214] This application embodiment also provides a sensor, which includes at least one processor and a communication interface. Optionally, the sensor further includes a memory, wherein the at least one processor is used to call a computer program stored in at least one memory, so that... Figure 3 The method and flow shown are thus implemented.

[0215] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program using computer program-related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing computer program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A self-motion estimation method, characterized in that, include: Obtain the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling According to the first rotational velocity vector ω s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e ,include: According to the second rotational velocity vector ω e The translation vector r determines the instantaneous velocity component caused by the rotation of the carrier; Based on the rotation matrix R and the first translational velocity vector Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; The second translational velocity vector t of the carrier is determined based on the instantaneous velocity component and the normalized instantaneous velocity vector. e The carrier is the carrier on which the sensor is located. The external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system, including the rotation matrix R and the translation vector r of the sensor coordinate system relative to the carrier coordinate system.

2. The method according to claim 1, characterized in that, According to the first rotational speed vector ω s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e ,include: Based on the rotation matrix R and the first rotational velocity vector ω s Determine the second rotational velocity vector ω of the carrier. e .

3. The method according to claim 1 or 2, characterized in that, The second translational velocity vector t of the carrier is determined based on the instantaneous velocity component and the normalized instantaneous velocity vector. e ,include: The amplitude of the translational velocity vector of the sensor coordinate system is determined based on the instantaneous velocity component and the normalized instantaneous velocity vector. The second translational velocity vector t of the carrier is determined based on the amplitude of the translational velocity vector in the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity components. e .

4. The method according to claim 3, characterized in that, The second translational velocity vector t of the carrier e , s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Based on the amplitude s of the translational velocity vector in the sensor coordinate system and the first translational velocity vector Determine the third translational velocity vector t of the sensor's self-motion. s .

6. The method according to claim 5, characterized in that, The sensor is a vision sensor, and the method further includes: Acquire the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the visual sensor; Based on the flow vector of the stationary obstacle and the third translational velocity vector t s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

7. A self-motion estimation device, characterized in that, include: The first acquisition unit is used to acquire the first rotational velocity vector ω of the sensor's self-motion. s The first translational velocity vector after normalization or scaling The first determining unit is configured to determine the first rotational velocity vector ω. s The first translational velocity vector The external parameters of the sensor determine the second rotational velocity vector ω of the carrier. e Second translational velocity vector t e This includes: based on the second rotational velocity vector ω e The instantaneous velocity component caused by the rotation of the carrier is determined by the translation vector r; based on the rotation matrix R and the first translational velocity vector... Determine the normalized instantaneous velocity vector of the sensor relative to the carrier coordinate system; determine the second translational velocity vector t of the carrier based on the instantaneous velocity components and the normalized instantaneous velocity vector. e The carrier is the carrier on which the sensor is located. The external parameters of the sensor include the transformation relationship between the sensor coordinate system and the carrier coordinate system, including the rotation matrix R and the translation vector r of the sensor coordinate system relative to the carrier coordinate system.

8. The apparatus according to claim 7, characterized in that, The first determining unit is configured to determine the rotation matrix R and the first rotational velocity vector ω. s Determine the second rotational velocity vector ω of the carrier. e .

9. The apparatus according to claim 7 or 8, characterized in that, The first determining unit is configured to determine the amplitude of the translational velocity vector of the sensor coordinate system based on the instantaneous velocity component and the normalized instantaneous velocity vector; and to determine the second translational velocity vector t of the carrier based on the amplitude of the translational velocity vector of the sensor coordinate system, the normalized instantaneous velocity vector, and the instantaneous velocity component. e .

10. The apparatus according to claim 9, characterized in that, The second translational velocity vector t of the carrier e , s is the magnitude of the translational velocity vector of the sensor coordinate system. For the normalized instantaneous velocity vector, t R The instantaneous velocity component is referred to here.

11. The apparatus according to claim 9 or 10, characterized in that, The device further includes: The second determining unit is configured to determine the magnitude s of the translational velocity vector of the sensor coordinate system and the first translational velocity vector based on the first translational velocity vector. Determine the third translational velocity vector t of the sensor's self-motion. s .

12. The apparatus according to claim 11, characterized in that, The sensor is a vision sensor, and the device further includes: The second acquisition unit is used to acquire the flow vector of a stationary obstacle, the flow vector including the motion vector of the stationary obstacle on the image plane of the visual sensor; The third determining unit is used to determine the flow vector of the stationary obstacle and the third translational velocity vector t. s and the first rotational velocity vector ω s Determine the depth z of the stationary obstacle.

13. A chip system, characterized in that, The chip system includes at least one processor and a communication interface, wherein the at least one processor is used to call a computer program stored in at least one memory to enable the device in which the chip system is located to implement the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on one or more processors, performs the method as described in any one of claims 1-6.

Images