Method for pose estimation and related device
Through the data fusion and feature enhancement of DVS and single-line lidar, the problem of low position estimation accuracy in complex environments is solved, high-precision position estimation in light-changing environments is achieved, and the power consumption of mobile devices is optimized and battery life.
Patent Information
- Application Number
- CN202310659950.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
Existing pose estimation methods have low accuracy in complex environments (such as excessive light or dimness), especially based on RGB color sensors and IMU methods, making it difficult to accurately estimate poses without environmental prior information.
Dynamic vision sensors (DVS) and single-line lidar are used to obtain perceptual data, and through data fusion and feature enhancement, the radar is not affected by light intensity and the characteristics of DVS to capture dynamic change information to improve the accuracy of position estimation.
In complex environments, the positioning can be accurately determined, the accuracy of positioning estimation can be improved, and the detection frequency of radar and DVS can be adjusted to balance accuracy and power consumption, reducing the power consumption of mobile devices and improving battery life.
Smart Images

Figure CN119087445A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle technology, and in particular to a method and related device for posture estimation. Background Art
[0002] With the development of science and technology, posture estimation technology is gradually applied in people's life, work and travel. Posture estimation technology refers to the technology of mobile devices (such as cars, robots, etc.) estimating their own posture in motion without prior information about the environment.
[0003] Currently, a known method for pose estimation is to extract feature points based on the information obtained by a red (red, R), green (green, G), and blue (blue, B) (RGB) color sensor and an inertial measurement unit (IMU), and then perform pose estimation based on the feature points. However, this method of pose estimation has low pose estimation accuracy in complex environments (such as strong light or dim light). Summary of the invention
[0004] The present application provides a posture estimation method and related devices to improve the accuracy of posture estimation.
[0005] In the first aspect, the present application provides a method for posture estimation, which can be executed by a mobile device, or the method can be executed by a component configured in the mobile device (such as a chip, a chip system, etc.), or it can be implemented by a logic module or software that can realize all or part of the functions of the mobile device. The present application does not limit this.
[0006] Exemplarily, the method includes: acquiring first perception data and second perception data, the first perception data being obtained by detecting the physical space where the mobile device is located through a dynamic vision sensor (DVS) of the mobile device, and the second perception data being obtained by detecting the physical space where the mobile device is located through a single-line laser radar of the mobile device; based on the first perception data, performing feature enhancement on first depth information to obtain second depth information, the first depth information being obtained by fusing the first perception data and the second perception data; and determining the posture of the mobile device based on the second depth information.
[0007] Based on the above scheme, the characteristic of radar that it is not affected by the intensity of light in the environment and the characteristic of DVS that it is good at capturing information of dynamic changes in the environment are utilized. By fusing the perception data obtained by DVS and the single-line laser radar and performing feature enhancement, the perception data obtained by DVS and the perception data obtained by the single-line laser radar complement each other. Whether in an environment with too strong light or in an environment with dim light, the position and posture of the mobile device can be accurately determined. In other words, in a complex environment, the accuracy of the position and posture estimation can be improved.
[0008] In combination with the first aspect, in some possible implementations, the first depth information includes multiple depth values, and based on the first perception data, the first depth information is feature enhanced to obtain the second depth information, including: determining the weight of each depth value in the multiple depth values based on the first perception data; and performing feature enhancement on the multiple depth values based on the weight of each depth value in the multiple depth values to obtain the second depth information.
[0009] DVS can capture moving feature points, that is, based on the first perception data obtained by DVS, the moving points and edge points can be highlighted. In the pose estimation method, the edge points and moving points are more important to the accuracy of pose estimation. Therefore, the weights of multiple depth values in the first depth information can be determined based on the first perception data to enhance the importance of moving points and edge points. Relatively speaking, this weakens the importance of static points. It can be understood that the second depth information is the depth information after feature enhancement of edge points and moving points based on the first depth information. It is conducive to the subsequent extraction and alignment of key features, which in turn helps to improve the accuracy of pose estimation.
[0010] In combination with the first aspect, in some possible implementations, the first perception data is one of one or more groups of first-category perception data obtained by the DVS from detecting the physical space where the mobile device is located, and the second perception data is one of one or more groups of second-category perception data obtained by the single-line laser radar from detecting the physical space where the mobile device is located; determining the posture of the mobile device based on the second depth information includes: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining the posture of the mobile device based on the second depth information; wherein the key depth information is second depth information whose difference with a previous group of second depth information is greater than a first threshold, the previous group of second depth information is obtained by feature enhancement of a previous group of first depth information based on a previous group of first perception data, the previous group of first depth information is obtained by fusing the previous group of first perception data and the previous group of second perception data, the ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference between the pixel values of the key feature points and the surrounding feature points is greater than or equal to the fourth threshold.
[0011] In combination with the first aspect, in some possible implementations, when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining the posture of the mobile device based on the second depth information includes: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, obtaining image-level posture information based on the second depth information, the image-level posture information being conversion information between the second depth information and a previous set of effective depth information or rich depth information, the previous set of effective depth information or rich depth information being determined based on the previous set of second depth information; obtaining feature-level position information based on the image-level pose information, the feature-level position information being position information after feature alignment of feature points in the second depth information based on the feature points in the previous set of effective depth information or rich depth information; and determining the posture of the mobile device based on the feature-level position information.
[0012] In combination with the first aspect, in some possible implementations, the method may also include: when the second depth information is key depth information and the second depth information is invalid depth information, increasing the frequency of detection of the physical space where the mobile device is located by the single-line laser radar and / or the DVS, the ratio of the number of key feature points included in the invalid depth information to the total number of feature points is less than a second threshold, and the key feature point is a feature point whose average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
[0013] When the second depth information is key depth information and invalid depth information, it can be considered that the richness of the perception data obtained based on the current frequency of the single-line laser radar and DVS is insufficient, and analysis and processing based on these data may not achieve the expected accuracy of pose estimation. In this case, the mobile device can increase the frequency of the single-line laser radar and / or DVS detecting the physical space where the mobile device is located to obtain richer perception data, which helps to improve the accuracy of pose estimation.
[0014] In combination with the first aspect, in some possible implementations, the method may also include: when the second depth information is key depth information and the second depth information is rich depth information, reducing the frequency of detection of the physical space where the mobile device is located by the single-line laser radar and / or the DVS, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than a third threshold, and the key feature point is a feature point whose average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
[0015] When the second depth information is the key depth information and the second depth information is rich depth information, it can be considered that the richness of the perception data obtained based on the current frequency of the single-line laser radar and DVS is relatively high, and analysis and processing based on these data can achieve the expected accuracy of posture estimation. However, when the frequency of detecting the physical space where the mobile device is located is too high, the power consumption of the single-line laser radar and DVS will also be relatively high, thereby increasing the power consumption of the mobile device. While ensuring good accuracy, the frequency of detecting the physical space where the mobile device is located by the single-line laser radar and / or DVS can be appropriately reduced, which is beneficial to reducing the power consumption of the mobile device and improving the battery life of the mobile device.
[0016] In the second aspect, the present application provides a mobile device, which includes a DVS, a single-line laser radar and a processor; wherein the DVS is used to: detect the physical space where the mobile device is located, and obtain first perception data; the single-line laser radar is used to: detect the physical space where the mobile device is located, and obtain second perception data; the processor is used to: based on the first perception data, perform feature enhancement on first depth information to obtain second depth information, and the first depth information is obtained by fusing the first perception data and the second perception data; and determine the posture of the mobile device based on the second depth information.
[0017] Based on the above scheme, the mobile device uses DVS and single-line laser radar to obtain perception data by utilizing the characteristics of radar being unaffected by the intensity of light in the environment and the characteristics of DVS being good at capturing dynamically changing information in the environment. By fusing the perception data obtained by DVS and single-line laser radar and performing feature enhancement, the perception data obtained by DVS and the perception data obtained by single-line laser radar complement each other, and the accuracy of pose estimation can be improved even in complex environments. Furthermore, single-line laser radar is cheaper than multi-line laser radar, and deploying single-line laser radar on mobile devices saves costs compared to deploying multi-line laser radar. Accordingly, the market price of mobile devices deployed with single-line laser radar is lower, so that mobile devices deployed with single-line laser radar can have a wider audience and better universality.
[0018] In combination with the second aspect, in some possible implementations, the first depth information includes multiple depth values, and the processor can be specifically used to: determine the weight of each depth value in the multiple depth values based on the first perception data; and perform feature enhancement on the multiple depth values based on the weight of each depth value in the multiple depth values to obtain the second depth information.
[0019] DVS can capture moving feature points, that is, based on the first perception data obtained by DVS, the moving points and edge points can be highlighted. In the pose estimation method, the edge points and moving points are more important to the accuracy of pose estimation. Therefore, the weights of multiple depth values in the first depth information can be determined based on the first perception data to enhance the importance of moving points and edge points. Relatively speaking, this weakens the importance of static points. It can be understood that the second depth information is the depth information after feature enhancement of edge points and moving points based on the first depth information. It is conducive to the subsequent extraction and alignment of key features, which in turn helps to improve the accuracy of pose estimation.
[0020] In combination with the second aspect, in some possible implementations, the first perception data is one of one or more groups of first-type perception data obtained by the DVS from detecting the physical space where the mobile device is located, and the second perception data is one of one or more groups of second-type perception data obtained by the single-line laser radar from detecting the physical space where the mobile device is located; the processor can be specifically used to: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determine the posture of the mobile device based on the second depth information; wherein the key depth information is second depth information whose difference with the previous group of second depth information is greater than a first threshold, the previous group of second depth information is obtained by feature enhancement of the previous group of first depth information based on the previous group of first perception data, the previous group of first depth information is obtained by fusing the previous group of first perception data and the previous group of second perception data, the ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference between the pixel values of the surrounding feature points is greater than or equal to the fourth threshold.
[0021] In combination with the second aspect, in some possible implementations, the processor may be specifically used to: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, obtain image-level pose information based on the second depth information, the image-level pose information being conversion information between the second depth information and a previous set of effective depth information or rich depth information, the previous set of effective depth information or rich depth information being determined based on the previous set of second depth information; obtain feature-level position information based on the image-level pose information, the feature-level position information being position information after feature alignment of feature points in the second depth information based on the feature points in the previous set of effective depth information or rich depth information; and determine the pose of the mobile device based on the feature-level position information.
[0022] In combination with the second aspect, in some possible implementations, the processor can also be used to: when the second depth information is key depth information and the second depth information is invalid depth information, send a first instruction to the single-line laser radar and / or DVS, the first instruction being used to instruct to increase the frequency of detection of the physical space where the mobile device is located, the ratio of the number of key feature points included in the invalid depth information to the total number of feature points is less than a second threshold, and the key feature point is a feature point whose average value of the difference in pixel values with the surrounding feature points is greater than or equal to a fourth threshold.
[0023] When the second depth information is key depth information and the second depth information is invalid depth information, it can be considered that the richness of the perception data obtained based on the current frequency of the single-line laser radar and DVS is insufficient, and analysis and processing based on these data may not achieve the expected accuracy of pose estimation. In this case, the processor can control the single-line laser radar and / or DVS to increase the frequency of detecting the physical space where the mobile device is located to obtain richer perception data, which helps to improve the accuracy of pose estimation.
[0024] In combination with the second aspect, in some possible implementations, the processor may also be used to: when the second depth information is key depth information and the second depth information is invalid depth information, send a second instruction to the single-line lidar and / or DVS, the second instruction being used to instruct to reduce the frequency of detecting the physical space where the mobile device is located, and the ratio of the number of key feature points included in the enriched depth information to the total number of feature points is greater than a third threshold, and the key feature point is a feature point whose average value of the difference in pixel values with the surrounding feature points is greater than or equal to a fourth threshold.
[0025] When the second depth information is the key depth information and the second depth information is rich depth information, it can be considered that the richness of the perception data obtained based on the current frequency of the single-line laser radar and DVS is relatively high, and analysis and processing based on these data can achieve the expected accuracy of posture estimation, but when the frequency of detection of the physical space where the mobile device is located is too high, the power consumption of the single-line laser radar and DVS will also be high, thereby increasing the power consumption of the mobile device. While ensuring good accuracy, the processor can control the single-line laser radar and / or DVS to appropriately reduce the frequency of detection of the physical space where the mobile device is located, which is beneficial to reduce the power consumption of the mobile device and improve the battery life of the mobile device.
[0026] In conjunction with the second aspect, in some possible implementations, the mobile device may further include a memory, where the memory is used to store the computer program called and executed by the processor.
[0027] Optionally, the processor is coupled to the memory.
[0028] In a third aspect, the present application provides a radar, which is used to: after receiving a first instruction, increase the frequency of detection of the physical space where the radar is located based on the first instruction; and / or, after receiving a second instruction, reduce the frequency of detection of the physical space where the radar is located based on the second instruction.
[0029] Increasing the frequency helps obtain richer perception data; decreasing the frequency helps reduce power consumption.
[0030] In a fourth aspect, the present application provides a DVS, which is used to: after receiving a first instruction, increase the frequency of detecting the physical space where the DVS is located based on the first instruction; and / or, after receiving a second instruction, reduce the frequency of detecting the physical space where the DVS is located based on the second instruction.
[0031] Increasing the frequency helps obtain richer perception data; decreasing the frequency helps reduce power consumption.
[0032] In a fifth aspect, the present application provides a mobile device, which can be used to implement the method in the first aspect and any possible implementation of the first aspect. The mobile device includes a corresponding module for executing the above method. The module included in the mobile device can be implemented by software and / or hardware.
[0033] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above-mentioned first aspect and any possible implementation method of the first aspect, for example, processing the perception data involved in the above-mentioned method, etc.
[0034] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.
[0035] The chip system may be composed of the chip, or may include the chip and other discrete devices.
[0036] In the seventh aspect, the present application provides a computer-readable storage medium, on which a computer program (also referred to as code, or instructions) is stored. When the computer program is run by a computer, the method in the above-mentioned first aspect and any possible implementation of the first aspect is executed.
[0037] In an eighth aspect, the present application provides a computer program product, comprising: a computer program (also referred to as code, or instruction), which, when executed, enables the method in the above-mentioned first aspect and any possible implementation manner of the first aspect to be executed.
[0038] It should be understood that the fifth to eighth aspects of the present application correspond to the technical solutions of the first and second aspects of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a schematic block diagram of a mobile device applicable to the method for posture estimation provided in an embodiment of the present application;
[0040] Figure 2 is a schematic flow chart of a method for posture estimation provided in an embodiment of the present application;
[0041] Figure 3 is a schematic diagram of feature points at different positions provided in an embodiment of the present application;
[0042] Figure 4 is a schematic diagram of image-level feature alignment provided by an embodiment of the present application;
[0043] Figure 5 is a schematic diagram of feature-level alignment provided in an embodiment of the present application;
[0044] Figure 6 is another schematic flow chart of a method for posture estimation provided in an embodiment of the present application;
[0045] Figure 7 It is another schematic block diagram of a mobile device suitable for the method for posture estimation provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The technical solution in this application will be described below in conjunction with the accompanying drawings.
[0047] In order to clearly describe the technical solution of the embodiment of the present application, the following explanation is first made.
[0048] First, in the embodiments of the present application, descriptions such as "when...", "under the circumstances of...", "if" and "if" all refer to the mobile device (e.g., vehicle) making corresponding processing under certain objective circumstances, and do not limit the time, nor do they require the device to make judgments when implementing, nor do they mean that there are other limitations. It should be noted that in the embodiments of the present application, the above descriptions can be replaced with each other, and there is no limitation on this.
[0049] Second, in the embodiments of the present application, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a device, system, product or equipment comprising a series of modules, modules or units is not necessarily limited to those modules, modules or units clearly listed, but may include other modules, modules or units that are not clearly listed or inherent to these devices, systems, products or equipment.
[0050] Third, in the embodiments of the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship, but does not exclude the situation where the associated objects before and after are in an "and" relationship. The specific meaning can be understood in conjunction with the context.
[0051] Fourth, in the embodiments of the present application, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0052] Fifth, in the embodiments of the present application, words such as "first", "second" and "third" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first depth information and the second depth information are used to distinguish between different depth information, the first threshold, the second threshold and the third threshold are used to distinguish between different thresholds, and the first instruction and the second instruction are used to distinguish between different instructions, and their order of precedence is not limited. Those skilled in the art can understand that words such as "first", "second" and "third" do not limit the quantity and execution order, and words such as "first", "second" and "third" do not necessarily limit them to be different.
[0053] Sixth, in the embodiments of the present application, preset can be understood as predefined, defined, predefined, stored, pre-stored, pre-negotiated, or pre-configured, etc.
[0054] The following is a brief description of the terms involved in this application.
[0055] 1. Depth information: It may refer to the depth of the image scene. The depth information may characterize the distance between the target and the sensing device, that is, the relative distance. The depth information may be characterized by a depth map. The depth map may refer to an image that uses the depth (distance) values from the visual sensor to each point in the scene as pixel values. The depth information (or depth map) may be obtained, for example, using a camera or a depth sensor (for example, including directly using active light technology (radar laser, structured light, etc.)), or using a binocular camera, a multi-eye camera, a binocular vision sensor or a multi-eye vision sensor, and the like, and using a binocular or multi-eye matching algorithm to obtain it. In other words, sensors and cameras such as millimeter-wave radars and lidars may obtain depth information. This application does not limit the specific method for obtaining depth information. In this application, depth information may also be referred to as a depth value.
[0056] 2. Pixel: A pixel is a small square in a digital sequence that represents an image (a small square is an image element). These small squares have a clear position and assigned values. In this application, the value corresponding to each small square can be recorded as a pixel value. The pixel value involved in this application can include the depth information corresponding to the pixel point.
[0057] 3. Posture: It can refer to position and posture (or orientation).
[0058] 4. Single-line laser radar: refers to a radar whose laser source emits a single-line beam, which can also be called a two-dimensional (2D) radar. Single-line laser radars can be divided into triangulation laser radars and time of flight (TOF) laser radars according to their principles.
[0059] With the development of science and technology, posture estimation technology is gradually applied in people's life, work and travel. Posture estimation technology refers to the technology of mobile devices (such as cars, robots, etc.) estimating their own posture in motion without prior information about the environment. At present, a known posture estimation method is to extract feature points based on the information obtained by RGB color sensors and IMU, and then perform posture estimation based on feature points. However, this posture estimation method has low accuracy in complex environments (such as strong light or dim light).
[0060] Based on the above problems, the present application provides a method and related devices for pose estimation, which utilize the characteristics of radar that it is not affected by the intensity of light in the environment, and the characteristics of DVS that it is good at capturing dynamically changing information in the environment. By fusing the perception data obtained by DVS and 2D radar and performing feature enhancement, the pose is estimated based on the feature-enhanced data, in order to overcome the impact of the complex environment and improve the accuracy of pose estimation.
[0061] Before describing in detail the method for posture estimation provided by the embodiment of the present application, Figure 1 The mobile device applicable to the embodiments of the present application is described exemplarily. The mobile device may include, but is not limited to, a car or a robot.
[0062] Figure 1 It is a schematic block diagram of a mobile device suitable for the method for posture estimation provided in an embodiment of the present application.
[0063] like Figure 1As shown, the mobile device 100 may include a 2D radar 101, a DVS 102, a feature fusion depth estimation module 103, a feature enhancement module 104, a feature extraction module 105, an image-level feature alignment module 106, a feature-level alignment module 107 and a result optimization module 108.
[0064] Among them, the 2D radar 101 and the DVS 102 can be used to obtain perception data. For example, the DVS 102 can be used to detect the physical space where the mobile device 100 is located to obtain first perception data (which can also be recorded as event data); the 2D radar can be used to detect the physical space where the mobile device 100 is located to obtain second perception data (which can also be recorded as radar data).
[0065] The feature fusion depth estimation module 103 can be used to fuse radar data and event data to obtain first depth information.
[0066] The feature enhancement module 104 may be configured to perform feature enhancement on the first depth information to obtain second depth information.
[0067] The feature extraction module 105 can be used to analyze the second depth information and determine whether the second depth information is key depth information. In the case where the second depth information is key depth information, the feature extraction module 105 can also determine the type corresponding to the second depth information. The type corresponding to the second depth information can include valid depth information, rich depth information, and invalid depth information.
[0068] The image-level feature alignment module 106 may be used to process the second depth information to obtain image-level pose information when the second depth information is key depth information and the second depth information is effective depth information or rich depth information.
[0069] The feature-level alignment module 107 may be used to obtain feature-level position information based on image-level pose information.
[0070] The result optimization module 108 can be used to obtain the posture information of the mobile device 100 based on the feature-level position information, that is, to determine the posture of the mobile device 100; the result optimization module 108 can also be used to obtain depth optimization information, and feed the depth optimization information back to the feature enhancement module 104, so that the subsequent feature enhancement module 104 can obtain more accurate second depth information.
[0071] In some embodiments, the mobile device 100 may further include a frequency adjustment module 109, which may be used to, when the second depth information is key depth information and the second depth information is invalid depth information, send a first instruction to the 2D radar to instruct the 2D optical radar to increase the frequency of detecting the physical space where the mobile device 100 is located, so that the 2D radar can obtain richer radar data, which helps to improve the accuracy of pose estimation; and / or, when the second depth information is key depth information and the second depth information is rich depth information, send a second instruction to the 2D radar to instruct the 2D optical radar to reduce the frequency of detecting the physical space where the mobile device 100 is located, so as to reduce the power consumption of the 2D radar.
[0072] The following combination Figure 2 The method for pose estimation provided in this application is described.
[0073] Figure 2 It is a schematic flowchart of a method for posture estimation provided in an embodiment of the present application.
[0074] like Figure 2 As shown, the method 200 may include steps 210 to 230. The steps of the method 200 may be performed by a mobile device configured with a 2D radar and a DVS (for ease of description, the mobile device configured with a 2D radar and a DVS is referred to as a mobile device in this application), or the method 200 may also be performed by a component configured in the mobile device (such as a chip, a chip system, etc.), or may also be implemented by a logic module or software capable of implementing all or part of the functions of the mobile device, which is not limited in the embodiments of the present application. As an example and not a limitation, the mobile device may include, but is not limited to, a robot, a vehicle, etc.
[0075] Below Figure 2 Detailed description of each step in.
[0076] In step 210, first perception data and second perception data are acquired.
[0077] The first perception data is obtained by detecting the physical space where the mobile device is located through the DVS of the mobile device; the second perception data is obtained by detecting the physical space where the mobile device is located through the 2D radar of the mobile device.
[0078] The mobile device can use DVS to detect the physical space where the mobile device is located to obtain the first perception data. It can be understood that DVS can use an event-driven approach to capture information about dynamic changes in the scene. During the continuous movement of the mobile device equipped with DVS, the objects in the physical space where the mobile device is located are constantly changing relative to the mobile device. DVS can capture the feature points of the movement, which can also be called events. Therefore, the first perception data can also be called event data.
[0079] As an example and not a limitation, an event may include (t, x, y, p), where "x, y" may represent the pixel coordinates of the event in 2D space, "t" may represent the timestamp of the event, and "p" may represent the polarity of the event. As an example and not a limitation, for example, when p=1, it may represent that the event is an acceleration event, that is, the characteristic point of the motion is the acceleration point; when p=-1, it may represent that the event is a deceleration event, that is, the characteristic point of the motion is the deceleration point; when p=0, it may represent that the event is a stationary event, that is, the characteristic point of the motion is the stationary point. This application does not make any limitation to this.
[0080] The mobile device may use a 2D radar to detect the physical space where the mobile device is located to obtain second perception data. The second perception data includes, but is not limited to, perception data obtained based on a triangulation laser radar or a TOF laser radar.
[0081] In a possible design, both the DVS and the 2D radar can obtain perception data based on a preset frequency. For example, the DVS can obtain the first perception data based on the first frequency, which can be, for example, a frequency preset when the DVS leaves the factory; the 2D radar can obtain the second perception data based on the second frequency, which can be, for example, a frequency preset when the 2D radar leaves the factory. The first frequency and the second frequency can be equal or unequal, and this application does not limit this. In this article, for the sake of ease of understanding and explanation, the perception data obtained by the DVS and the 2D radar each time based on their respective frequencies are recorded as a group of perception data. That is, the aforementioned first perception data can be one of one or more groups of perception data obtained by the DVS, and the second perception data can be one of one or more groups of perception data obtained by the 2D radar. For the sake of distinction, the perception data obtained by the DVS is recorded as the first type of perception data, and the perception data obtained by the 2D radar is recorded as the second type of perception data. Therefore, the first perception data can be one of one or more groups of first-type perception data, and the second perception data can be one of one or more groups of second-type perception data.
[0082] In step 220, based on the first perception data, features of the first depth information are enhanced to obtain second depth information, where the first depth information is obtained by fusing the first perception data and the second perception data.
[0083] After acquiring the first perception data and the second perception data, the mobile device may analyze and process the first perception data and the second perception data.
[0084] Exemplarily, the mobile device may first align timestamps of the first perception data and the second perception data.
[0085] As an example but not a limitation, performing timestamp alignment may include, but is not limited to, taking the perception data acquired by the sensor with a lower frequency as a reference, and for each timestamp in the perception data acquired by the sensor with a lower frequency, finding the data corresponding to the timestamp closest to each timestamp in the perception data acquired by other sensors. For example, when the first frequency is greater than the second frequency, the mobile device may take the second perception data acquired by the second frequency as a reference, and find the timestamp closest to each timestamp in the second perception data in the first perception data, and then find the data corresponding to these timestamps.
[0086] Secondly, the mobile device can fuse the data corresponding to these timestamps with the second perception data to obtain the first depth information.
[0087] It is understandable that, when the second frequency is greater than or equal to the first frequency, the mobile device may perform timestamp alignment based on the first sensing data acquired at the first frequency. For the sake of brevity, it will not be described in detail here.
[0088] After acquiring the first depth information, the mobile device may perform feature enhancement on the first depth information based on the first perception data to obtain second depth information.
[0089] In other words, the second depth information is information obtained by performing feature enhancement on the first depth information.
[0090] In some embodiments, the first depth information includes multiple depth values, and based on the first perception data, the first depth information is feature enhanced to obtain the second depth information, including: determining the weight of each depth value in the multiple depth values based on the first perception data; and performing feature enhancement on the multiple depth values based on the weight of each depth value in the multiple depth values to obtain the second depth information.
[0091] As mentioned above, DVS can capture moving feature points, that is, based on the first perception data obtained by DVS, the moving points and edge points can be highlighted, and in the pose estimation method, the edge points and the moving points are more important to the accuracy of the pose estimation. Therefore, the weights of the multiple depth values in the first depth information can be determined based on the first perception data to enhance the importance of the moving points and edge points. Relatively speaking, this weakens the importance of the static points.
[0092] As an example but not limitation, the mobile device can input the first perception data and the first depth information into the attention model, use the first perception data as attention, determine the weights of multiple depth values in the first depth information, and filter the first depth information based on the weights of the multiple depth values, so as to enhance the importance of moving points and edge points, and finally obtain the second depth information. It can be understood that the second depth information is the depth information after feature enhancement of edge points and moving points based on the first depth information. It is conducive to the subsequent extraction and alignment of key features, and thus helps to improve the accuracy of pose estimation.
[0093] As an example and not a limitation, in some possible implementations, the importance of moving points and edge points can be relatively enhanced by weakening the importance of static points. This application does not impose any limitation on this.
[0094] The present application does not limit the specific data format of the first depth information and the second depth information. As an example but not a limitation, the first depth information and the second depth information may be data in a matrix format.
[0095] In step 230 , the position and posture of the mobile device is determined based on the second depth information.
[0096] After obtaining the second depth information, the mobile device may analyze the second depth information to further determine the position and posture of the mobile device.
[0097] In some embodiments, determining the posture of the mobile device based on the second depth information includes: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining the posture of the mobile device based on the second depth information; wherein the key depth information is second depth information whose difference with a previous group of second depth information is greater than a first threshold, the previous group of second depth information is obtained by feature enhancement of a previous group of first depth information based on a previous group of first perception data, the previous group of first depth information is obtained by fusion of the previous group of first perception data and the previous group of second perception data, the ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference between the pixel values of the key feature points and the surrounding feature points is greater than or equal to the fourth threshold.
[0098] Among them, the first threshold, the second threshold, the third threshold and the fourth threshold may be pre-set thresholds, which are not limited in this application.
[0099] As mentioned above, both DVS and 2D radar can obtain perception data based on a preset frequency. The frequency at which the mobile device calculates the second depth information can be equal to the frequency corresponding to the sensor with the lower frequency in the DVS and 2D radar. For example, when the first frequency is greater than the second frequency, the mobile device can calculate a set of second depth information after each set of second perception data is obtained by the 2D radar.
[0100] After obtaining a set of second depth information, the mobile device may first determine whether the set of second depth information is key depth information. In a possible implementation, if the difference between the set of second depth information and the previous set of second depth information is greater than a first threshold, the set of second depth information may be determined as key depth information.
[0101] It can be understood that in actual application scenarios, the first set of second depth information calculated each time the mobile device is turned on can be defaulted to the key depth information, or the first set of second depth information calculated each time the mobile device is turned on can be defaulted to the non-key depth information, and the present application does not impose any limitations on this.
[0102] When this set of second depth information is key depth information, the mobile device may further determine whether this set of second depth information is valid depth information or rich depth information; when this set of second depth information is non-key depth information, the mobile device may not perform subsequent analysis and processing on this set of second depth information.
[0103] In the case where the set of second depth information is key depth information, the mobile device may first determine a key feature point based on the second depth information. A key feature point is a feature point whose average value of the difference between the pixel values of surrounding feature points is greater than or equal to a fourth threshold value, that is, when the average value of the difference between the pixel value of a feature point and the pixel values of surrounding feature points is greater than or equal to the fourth threshold value, the feature point may be determined as a key feature point.
[0104] In order to better understand the key feature points, the following Figure 3 Describe the key feature points.
[0105] Figure 3 It is a schematic diagram of feature points at different positions provided in an embodiment of the present application.
[0106] Taking the first feature point as an example, Figure 3 As shown, in the case where the first feature point is a corner point, for example, in the case where the first feature point is feature point 1, the feature points around it may include feature point 2, feature point 3 and feature point 4, and in the case where the average value of the difference in pixel values between feature point 1 and feature point 2, feature point 3 and feature point 4 is greater than or equal to the fourth threshold, feature point 1 is a key feature point; in the case where the average value of the difference in pixel values between feature point 1 and feature point 2, feature point 3 and feature point 4 is less than the fourth threshold, feature point 1 is not a key feature point.
[0107] When the first feature point is an edge point, for example, when the first feature point is feature point 5, the feature points around it may include feature point 3, feature point 4, feature point 6, feature point 7 and feature point 8. When the average value of the difference between the pixel values of feature point 5 and feature points 3, 4, 6, 7 and 8 is greater than or equal to the fourth threshold, feature point 5 is a key feature point; when the average value of the difference between the pixel values of feature point 5 and feature points 3, 4, 6, 7 and 8 is less than the fourth threshold, feature point 5 is not a key feature point.
[0108] When the first feature point is an intermediate point (that is, neither a corner point nor an edge point), for example, when the first feature point is feature point 9, the feature points around it may include feature point 3, feature point 6, feature point 7, feature point 10, feature point 11, feature point 12, feature point 13 and feature point 14. When the average value of the difference in pixel values between feature point 9 and feature point 3, feature point 6, feature point 7, feature point 10, feature point 11, feature point 12, feature point 13 and feature point 14 is greater than or equal to a fourth threshold, feature point 9 is a key feature point; when the average value of the difference in pixel values between feature point 9 and feature point 3, feature point 6, feature point 7, feature point 10, feature point 11, feature point 12, feature point 13 and feature point 14 is less than the fourth threshold, feature point 9 is not a key feature point.
[0109] After determining the key feature points, the mobile device may determine whether the second depth information is valid depth information or enriched depth information based on the number of the key feature points.
[0110] As an example but not limitation, when the ratio of the number of key feature points included in the second depth information to the total number of feature points included in the second depth information is greater than or equal to the second threshold and less than or equal to the third threshold, the second depth information can be determined as valid depth information; when the ratio of the number of key feature points included in the second depth information to the total number of feature points included in the second depth information is greater than the third threshold, the second depth information can be determined as rich depth information; when the ratio of the number of key feature points included in the second depth information to the total number of feature points included in the second depth information is less than the second threshold, the second depth information can be determined as invalid depth information.
[0111] When the second depth information is valid depth information or rich depth information, the mobile device may determine the posture of the mobile device based on the second depth information; when the second depth information is invalid depth information, the mobile device may not perform subsequent analysis and processing on the second depth information.
[0112] In the present application, a feature point may include at least one pixel point.
[0113] In some embodiments, when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining the posture of the mobile device based on the second depth information includes: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, obtaining image-level posture information based on the second depth information, the image-level posture information being conversion information between the second depth information and a previous set of effective depth information or rich depth information, the previous set of effective depth information or rich depth information being determined based on the previous set of second depth information; obtaining feature-level position information based on the image-level pose information, the feature-level position information being position information after feature alignment of feature points in the second depth information based on the feature points in the previous set of effective depth information or rich depth information; and determining the posture of the mobile device based on the feature-level position information.
[0114] Figure 4 It is a schematic diagram of image-level feature alignment provided in an embodiment of the present application.
[0115] For example, Figure 4 As shown, I k,c The second depth information can be represented by I k-1,c The previous set of effective depth information or rich depth information that can represent the second depth information, T k,k-1 It can represent the image-level pose information, that is, the second depth information I k,c Together with the previous set of effective depth information or rich depth information I k-1,c That is, in the second depth information I k,c is the key depth information, and the second depth information is the effective depth information or the rich depth information, the mobile device may, based on the second depth information I k,c Get image-level pose information T k,k-1 .
[0116] Optionally, T k,k-1 Can satisfy: (For ease of description, this formula is recorded as Formula 1).
[0117] In formula 1, π can represent the symbol of world coordinates to pixel coordinates, π -1 It can represent the conversion of pixel coordinates to world coordinate symbols, u can represent the location information of the feature point (such as pixel coordinates), D u It can represent the depth value corresponding to u, and ∫∫...du is the double integral symbol.
[0118] After obtaining the image-level pose information, the mobile device can obtain feature-level position information based on the image-level pose information.
[0119] It can be understood that in actual application scenarios, each time the mobile device is turned on, the first group of second depth information calculated is used as the key depth information by default. If the first group of second depth information is valid depth information or rich depth information, the previous group of valid depth information or rich depth information of the first group of second depth information can be preset information, and the present application does not impose any limitations on this.
[0120] Figure 5 It is a schematic diagram of feature-level alignment provided in an embodiment of the present application.
[0121] For example, Figure 5 As shown, I k,c The second depth information can be represented by I k-1,c The previous set of valid depth information or enriched depth information may represent the second depth information, u i ′ can be expressed in I k,c The position information of the i-th feature point in i It can be expressed in I k-1,c Zhong and I k,c The position information of the feature point corresponding to the i-th feature point, u i ″ can represent feature-level position information, that is, I k,c The i-th feature point in is based on u i That is, in the second depth information I k,c is the key depth information, and the second depth information is the effective depth information or the rich depth information, the mobile device may k,k-1 、u i and u i ′Get feature-level position information u i ″.
[0122] Optionally, u i "Can meet: (For ease of description, this formula is recorded as Formula 2).
[0123] In formula 2, P k,c (u i ′) can be expressed in I k,c China-Israel i The feature block is centered on the feature point corresponding to P ′, and the feature block may include N×N feature points, where N is an odd number greater than 0; k-1,c (u i ) can be expressed in I k-1,c China-Israel i The feature block centered on the corresponding feature point may also include N×N feature points; T k,k-1 Can represent I k-1,c with Ik,c The conversion information between them is also the above-mentioned image-level pose information.
[0124] After obtaining the feature-level position information, the mobile device may determine the position and posture information of the mobile device based on the feature-level position information.
[0125] The mobile device can obtain the conversion information T between the pixel coordinate system and the world coordinate system based on the feature-level position information k,w , and then based on T k,w A pose of the mobile device is determined.
[0126] Optionally, T k,w Can satisfy: (For ease of description, this formula is recorded as Formula 3).
[0127] In formula 3, u i ″ can represent feature-level position information, π can represent the symbol of world coordinate conversion to pixel coordinate, Can represent u i ″The position information of the corresponding feature point in the world coordinate system.
[0128] Based on T k,w The change information of the position and posture compared with the movement and transposition determined based on the previous set of second depth information can be known, and the current position and posture information of the mobile device can be obtained.
[0129] In a possible implementation, the mobile device may also be based on the feature-level location information u i ″ and the second depth information to obtain the third depth information, that is, Figure 1 It can be understood that, compared with the second depth information, the depth value of the feature point included in the third depth information remains unchanged, but the position information of the feature point is changed from u i ' becomes u i ".like Figure 1 As shown, after the result optimization module 108 obtains the third depth information, the third depth information can be fed back to the feature enhancement module 104, so that the feature enhancement module 104 can be more accurate in the subsequent calculation of the second depth information. That is, as the number of iterations of the feature enhancement module 104 increases, the second depth information determined by the feature enhancement module 104 will be closer and closer to the true value, thereby helping to further improve the accuracy of the pose estimation.
[0130] In some embodiments, method 200 may further include: when the second depth information is key depth information and the second depth information is invalid depth information, increasing the frequency of the single-line laser radar detecting the physical space where the mobile device is located.
[0131] Among them, the ratio of the number of key feature points included in the invalid depth information to the total number of feature points is less than the second threshold, that is, when the ratio of the number of key feature points included in the second depth information to the total number of feature points included in the second depth information is less than the second threshold, the second depth information can be determined as invalid depth information. The key feature point is a feature point whose average value of the difference between the pixel values of the surrounding feature points is greater than or equal to the fourth threshold.
[0132] When the second depth information is key depth information and invalid depth information, it can be considered that the richness of the perception data obtained by the 2D radar based on the current frequency is insufficient, and analysis and processing based on these data may not achieve the expected accuracy of pose estimation. In this case, the mobile device can increase the frequency of the 2D radar detecting the physical space where the mobile device is located, so that the 2D radar can obtain richer perception data, which helps to improve the accuracy of pose estimation.
[0133] In some embodiments, method 200 may further include: when the second depth information is key depth information and the second depth information is rich depth information, reducing the frequency of the single-line laser radar detecting the physical space where the mobile device is located.
[0134] Among them, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than the third threshold, that is, when the ratio of the number of key feature points included in the second depth information to the total number of feature points included in the second depth information is greater than the third threshold, the second depth information can be determined as rich depth information. The key feature point is a feature point whose average value of the difference between the pixel values of the surrounding feature points is greater than or equal to the fourth threshold.
[0135] When the second depth information is the key depth information and the second depth information is rich depth information, it can be considered that the richness of the perception data obtained by the 2D radar based on the current frequency is relatively high, and analysis and processing based on these data can achieve the expected accuracy of posture estimation. However, when the frequency of the 2D radar detecting the physical space where the mobile device is located is too high, the power consumption of the 2D radar will also be high, thereby increasing the power consumption of the mobile device. While ensuring good accuracy, the frequency of the 2D radar detecting the physical space where the mobile device is located can be appropriately reduced, which is beneficial to reducing the power consumption of the mobile device and improving the battery life of the mobile device.
[0136] Similar to adjusting the frequency of the 2D radar, in some possible implementations, when the second depth information is key depth information and the second depth information is invalid depth information, the frequency of DVS detecting the physical space where the mobile device is located can be increased; and when the second depth information is key depth information and the second depth information is rich depth information, the frequency of DVS detecting the physical space where the mobile device is located can be reduced.
[0137] It should be understood that the key depth information, effective depth information, rich depth information and invalid depth information defined herein are only for distinguishing different depth information, and the present application does not limit the naming of these depth information. In addition, the key depth information, effective depth information, rich depth information and invalid depth information described above in combination with the first threshold, the second threshold, the third threshold and the fourth threshold are only one possible design, and those skilled in the art can also make simple changes based on the same concept to obtain other possible definitions of key depth information, effective depth information, rich depth information and invalid depth information, which will not be elaborated here.
[0138] In order to better facilitate the method for posture estimation provided by this application, the following Figure 1 and Figure 6 The method for posture estimation provided in this application is further explained.
[0139] Figure 6 It is another schematic flowchart of the method for posture estimation provided in an embodiment of the present application.
[0140] In step 601, a feature fusion depth estimation module is used to obtain conversion information between the coordinates of the DVS and the 2D radar in response to a user's calibration operation on the DVS and the 2D radar.
[0141] Mobile devices can use Figure 1 The feature fusion depth estimation module 103 shown obtains the conversion information between the coordinates of the DVS and the 2D radar in response to the user's calibration operation on the DVS and the 2D radar. That is, the feature fusion depth estimation module 103 can be used to obtain the conversion information between the coordinates of the DVS and the 2D radar in response to the user's calibration operation on the DVS and the 2D radar.
[0142] In step 602, a feature fusion depth estimation module is used to obtain first depth information based on the first perception data obtained by the DVS and the second perception data obtained by the 2D radar.
[0143] Mobile devices can use Figure 1 The 2D radar 101 shown in the figure obtains the second perception data, and the DVS 102 obtains the first perception data. The mobile device can also use Figure 1The feature fusion depth estimation module 103 shown calculates the first depth information based on the first perception data and the second perception data and based on the conversion information between the coordinates of the DVS and the 2D radar obtained in step 601. For a detailed description of the first depth information, please refer to the relevant content above, which will not be repeated here for brevity.
[0144] In step 603, a feature enhancement module is used to obtain second depth information based on the first depth information.
[0145] Mobile devices can use Figure 1 The feature enhancement module 104 shown performs feature enhancement on the first depth information based on the first perception data to obtain the second depth information. For a detailed description, please refer to the relevant description above. For the sake of brevity, it will not be repeated here.
[0146] In step 604, a feature extraction module is used to extract key feature points when the second depth information is key depth information.
[0147] Mobile devices can use Figure 1 The feature extraction module 105 shown first determines whether the second depth information is key depth information, and extracts key feature points when the second depth information is key depth information. For a detailed description of determining whether the second depth information is key depth information and extracting key feature points (i.e., determining key feature points), please refer to the relevant content above, which will not be repeated here for brevity.
[0148] The mobile device may also use the feature extraction module 105 to determine the type corresponding to the second depth information based on the number of key feature points. As mentioned above, the type corresponding to the second depth information may include valid depth information, rich depth information, and invalid depth information. For a detailed description of valid depth information, rich depth information, and invalid depth information, please refer to the relevant content above, and for the sake of brevity, it will not be repeated here.
[0149] In step 605, an image-level feature alignment module is used to obtain image-level pose information.
[0150] Mobile devices can use Figure 1 The image-level feature alignment module 106 shown obtains image-level pose information based on the second depth information when the second depth information is key depth information and the second depth information is effective depth information or rich depth information. For a detailed description, please refer to the relevant content above. For the sake of brevity, it will not be repeated here.
[0151] In step 606, a feature-level alignment module is used to perform feature alignment based on the image-level pose information to obtain feature-level position information.
[0152] Mobile devices can use Figure 1 The feature-level alignment module 107 shown performs feature alignment based on image-level pose information to obtain feature-level position information. For a detailed description, please refer to the relevant content above. For the sake of brevity, it will not be repeated here.
[0153] In step 607, the pose of the mobile device is determined based on the feature-level position information using the result optimization module.
[0154] Mobile devices can use Figure 1 The result optimization module 108 shown optimizes the conversion information between the world coordinate system and the pixel coordinate system based on the feature-level position information, and then determines the position and posture of the mobile device. For detailed description, please refer to the relevant content above. For the sake of brevity, it will not be repeated here.
[0155] Based on the above scheme, the characteristic of radar that it is not affected by the intensity of light in the environment and the characteristic of DVS that it is good at capturing the information of dynamic changes in the environment are utilized. By fusing the perception data obtained by DVS and 2D radar and performing feature enhancement, the perception data obtained by DVS and the perception data obtained by 2D radar complement each other, and the accuracy of pose estimation can be improved even in a complex environment. In addition, the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located can be adjusted according to the richness of the acquired perception data. For example, when the richness of the acquired perception data is insufficient, the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located can be increased, so as to obtain more abundant perception data in the future, thereby helping to achieve the expected accuracy of pose estimation; for another example, when the richness of the acquired perception data is excessive, the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located can be reduced, so as to reduce the power consumption of the mobile device and improve the endurance of the mobile device.
[0156] The present application also provides a mobile device, which includes a DVS, a 2D radar and a processor; wherein the DVS and the 2D radar are used to obtain perception data, and the processor is used to call and execute a computer program so that the mobile device performs the above Figure 2 or Figure 6 The steps involved in any of the embodiments described herein.
[0157] Figure 7 It is another schematic block diagram of a mobile device suitable for the method for posture estimation provided in an embodiment of the present application.
[0158] like Figure 7The mobile device 700 shown may include a 2D radar 710 , a DVS 720 , and a processor 730 .
[0159] For example, DVS 720 can be used to: detect the physical space where the mobile device is located to obtain first perception data; the 2D radar 710 can be used to: detect the physical space where the mobile device is located to obtain second perception data; the processor 730 can be used to: based on the first perception data, perform feature enhancement on the first depth information to obtain second depth information, the first depth information is obtained by fusing the first perception data and the second perception data; determine the posture of the mobile device 700 based on the second depth information.
[0160] For a detailed description, please refer to the above Figure 2 or Figure 6 A related description of the steps involved in any of the embodiments described.
[0161] In some possible implementations, the first depth information includes multiple depth values, and the processor 730 can be specifically used to: determine the weight of each depth value in the multiple depth values based on the first perception data; and enhance the features of the multiple depth values based on the weight of each depth value in the multiple depth values to obtain the second depth information.
[0162] In some possible implementations, the first perception data is the DVS 720 is one of the one or more groups of first-category perception data obtained by detecting the physical space where the mobile device is located, and the second perception data is one of the one or more groups of second-category perception data obtained by the single-line laser radar detecting the physical space where the mobile device is located; the processor 730 can be specifically used to: when the second depth information is key depth information, and the second depth information is effective depth information or rich depth information, determine the posture of the mobile device 700 based on the second depth information; wherein the key depth information is the second depth information whose difference with the previous group of second depth information is greater than the first threshold, the previous group of second depth information is obtained by feature enhancement of the previous group of first depth information based on the previous group of first perception data, the previous group of first depth information is obtained by fusing the previous group of first perception data and the previous group of second perception data, the ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference between the pixel values of the surrounding feature points is greater than or equal to the fourth threshold.
[0163] In some possible implementations, the processor 730 may be specifically used to: when the second depth information is key depth information and the second depth information is effective depth information or rich depth information, obtain image-level pose information based on the second depth information, the image-level pose information being conversion information between the second depth information and a previous set of effective depth information or rich depth information, the previous set of effective depth information or rich depth information being determined based on the previous set of second depth information; obtain feature-level position information based on the image-level pose information, the feature-level position information being position information after feature alignment of feature points in the second depth information based on the feature points in the previous set of effective depth information or rich depth information; and determine the pose of the mobile device 700 based on the feature-level position information.
[0164] In some possible implementations, the processor 730 may also be used to: when the second depth information is key depth information and the second depth information is invalid depth information, send a first instruction to the 2D radar 710 and / or DVS 720, the first instruction being used to instruct to increase the frequency of detecting the physical space where the mobile device 700 is located, the ratio of the number of key feature points included in the invalid depth information to the total number of feature points being less than a second threshold, and the key feature point being a feature point for which the average value of the difference in pixel values with the surrounding feature points is greater than or equal to a fourth threshold.
[0165] In some possible implementations, the processor 730 may also be used to: when the second depth information is key depth information and the second depth information is invalid depth information, send a second instruction to the 2D radar 710 and / or DVS 720, the second instruction being used to instruct to reduce the frequency of detecting the physical space where the mobile device 700 is located, the ratio of the number of key feature points included in the enriched depth information to the total number of feature points being greater than a third threshold, and the key feature point being a feature point for which the average value of the difference in pixel values with the surrounding feature points is greater than or equal to a fourth threshold.
[0166] For a detailed description of the above possible implementation methods, please refer to the relevant content above. For the sake of brevity, they will not be repeated here.
[0167] It should be understood that the mobile devices involved in the present application may include but are not limited to vehicles, robots (such as sweeping robots, food delivery robots, etc.), etc.
[0168] In a possible implementation, the mobile device 700 may further include a memory, where the memory is used to store a computer program called and executed by the processor 730 .
[0169] Optionally, the processor 730 is coupled to a memory.
[0170] Based on the above scheme, the mobile device uses DVS and 2D radar to obtain perception data by utilizing the characteristic that radar is not affected by the intensity of light in the environment and the characteristic that DVS is good at capturing dynamically changing information in the environment. By fusing the perception data obtained by DVS and 2D radar and performing feature enhancement, the perception data obtained by DVS and the perception data obtained by 2D radar complement each other, and the accuracy of pose estimation can be improved even in a complex environment. In addition, the mobile device can also adjust the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located according to the richness of the acquired perception data. For example, when the richness of the acquired perception data is insufficient, the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located can be increased, so as to obtain more abundant perception data in the future, thereby helping to achieve the expected accuracy of pose estimation; for another example, when the richness of the acquired perception data is excessive, the frequency of 2D radar and / or DVS detecting the physical space where the mobile device is located can be reduced, so as to reduce the power consumption of the mobile device and improve the endurance of the mobile device. Furthermore, 2D radar is cheaper than multi-line lidar. Deploying 2D radar on mobile devices saves costs compared to deploying multi-line lidar. Correspondingly, the market price of mobile devices deployed with 2D radar is lower. Therefore, mobile devices deployed with 2D radar can have a wider audience and better universality.
[0171] The present application also provides a radar, which is used to: adjust the frequency of detecting the physical space where the radar is located in response to a received frequency adjustment instruction.
[0172] For example, the frequency adjustment instruction includes a first instruction and / or a second instruction. The radar can be used to: after receiving the first instruction, increase the frequency of detecting the physical space where the radar is located based on the first instruction; and / or, after receiving the second instruction, reduce the frequency of detecting the physical space where the radar is located based on the second instruction.
[0173] Optionally, the radar may be a single-line laser radar or a multi-line laser radar.
[0174] The present application also provides a DVS, which is used to: in response to a received frequency adjustment instruction, adjust the frequency of detecting a physical space where the DVS is located.
[0175] For example, the frequency adjustment instruction includes a first instruction and / or a second instruction. The DVS may be configured to: after receiving the first instruction, increase the frequency of detecting the physical space where the DVS is located based on the first instruction; and / or, after receiving the second instruction, reduce the frequency of detecting the physical space where the DVS is located based on the second instruction.
[0176] The present application also provides a chip system, which includes a processor for implementing the above Figure 2 or Figure 6 The functions involved in the steps of moving the device in any embodiment.
[0177] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.
[0178] The chip system may be composed of the chip, or may include the chip and other discrete devices.
[0179] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, the computer executes the above Figure 2 or Figure 6 The step of moving the device in any embodiment.
[0180] The present application also provides a computer program product, including a computer program. When the computer program is executed, the computer executes the above Figure 2 or Figure 6 The step of moving the device in any embodiment.
[0181] It should be understood that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined to perform. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0182] It should also be understood that the memory in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be a ROM, a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable programmable read-only memory (electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (doubledata rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM). It should be noted that memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
[0183] The terms "unit", "module" and the like used in this specification may be used to represent a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution.
[0184] It will be appreciated by those skilled in the art that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in combination with computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0185] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0186] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more units may be integrated into one module.
[0187] In the above embodiments, the functions of each functional module can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).
[0188] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for pose estimation, characterized in that: The method comprises: Acquire first perception data and second perception data, wherein the first perception data is obtained by detecting the physical space where the mobile device is located through a dynamic visual sensor of the mobile device, and the second perception data is obtained by detecting the physical space where the mobile device is located through a single-line laser radar of the mobile device; Based on the first perception data, feature enhancement is performed on the first depth information to obtain second depth information, where the first depth information is obtained by fusing the first perception data and the second perception data; The pose of the mobile device is determined based on the second depth information.
2. The method according to claim 1, characterized in that The first depth information includes a plurality of depth values, and the step of performing feature enhancement on the first depth information based on the first perception data to obtain the second depth information includes: determining a weight for each of the plurality of depth values based on the first perception data; Feature enhancement is performed on the multiple depth values based on a weight of each depth value in the multiple depth values to obtain the second depth information.
3. The method according to claim 1 or 2, characterized in that The first perception data is one of one or more groups of first-category perception data obtained by the dynamic visual sensor detecting the physical space where the mobile device is located, and the second perception data is one of one or more groups of second-category perception data obtained by the single-line laser radar detecting the physical space where the mobile device is located; Determining the position and posture of the mobile device based on the second depth information includes: When the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining a position and posture of the mobile device based on the second depth information; Among them, the key depth information is second depth information whose difference with the previous group of second depth information is greater than the first threshold, the previous group of second depth information is obtained by feature enhancement of the previous group of first depth information based on the previous group of first perception data, and the previous group of first depth information is obtained by fusion of the previous group of first perception data and the previous group of second perception data. The ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the enriched depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference in pixel values with the surrounding feature points is greater than or equal to the fourth threshold.
4. The method according to claim 3, characterized in that When the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining the position and posture of the mobile device based on the second depth information includes: When the second depth information is key depth information and the second depth information is effective depth information or enriched depth information, image-level pose information is obtained based on the second depth information, where the image-level pose information is conversion information between the second depth information and a previous set of effective depth information or enriched depth information, where the previous set of effective depth information or enriched depth information is determined based on the previous set of second depth information; Obtain feature-level position information based on the image-level pose information, where the feature-level position information is position information after feature alignment of feature points in the second depth information based on feature points in the previous set of valid depth information or enriched depth information; The pose of the mobile device is determined based on the feature-level position information.
5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When the second depth information is key depth information and the second depth information is invalid depth information, the frequency of detection of the physical space where the mobile device is located by the single-line laser radar and / or the dynamic vision sensor is increased, the ratio of the number of key feature points included in the invalid depth information to the total number of feature points is less than a second threshold, and the key feature point is a feature point whose average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the second depth information is key depth information and the second depth information is rich depth information, the frequency of detection of the physical space where the mobile device is located by the single-line laser radar and / or the dynamic vision sensor is reduced, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than a third threshold, and the key feature point is a feature point for which the average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
7. A mobile device, characterized in that: It includes a dynamic vision sensor, a single-line laser radar and a processor; among which, The dynamic vision sensor is used to: detect the physical space where the mobile device is located to obtain first perception data; The single-line laser radar is used to detect the physical space where the mobile device is located to obtain second perception data; The processor is used to: Based on the first perception data, feature enhancement is performed on the first depth information to obtain second depth information, where the first depth information is obtained by fusing the first perception data and the second perception data; The pose of the mobile device is determined based on the second depth information.
8. The device according to claim 7, characterized in that The first depth information includes a plurality of depth values, and the processor is specifically configured to: determining a weight for each of the plurality of depth values based on the first perception data; Feature enhancement is performed on the multiple depth values based on a weight of each depth value in the multiple depth values to obtain the second depth information.
9. The device according to claim 7 or 8, characterized in that The first perception data is one of one or more groups of first-category perception data obtained by the dynamic visual sensor detecting the physical space where the mobile device is located, and the second perception data is one of one or more groups of second-category perception data obtained by the single-line laser radar detecting the physical space where the mobile device is located; The processor is specifically used for: When the second depth information is key depth information and the second depth information is effective depth information or rich depth information, determining a position and posture of the mobile device based on the second depth information; Among them, the key depth information is second depth information whose difference with the previous group of second depth information is greater than the first threshold, the previous group of second depth information is obtained by feature enhancement of the previous group of first depth information based on the previous group of first perception data, and the previous group of first depth information is obtained by fusion of the previous group of first perception data and the previous group of second perception data. The ratio of the number of key feature points included in the effective depth information to the total number of feature points is greater than or equal to the second threshold and less than or equal to the third threshold, the ratio of the number of key feature points included in the enriched depth information to the total number of feature points is greater than the third threshold, and the key feature point is a feature point whose average value of the difference in pixel values with the surrounding feature points is greater than or equal to the fourth threshold.
10. The device according to claim 9, characterized in that The processor is specifically used for: When the second depth information is key depth information and the second depth information is effective depth information or enriched depth information, image-level pose information is obtained based on the second depth information, where the image-level pose information is conversion information between the second depth information and a previous set of effective depth information or enriched depth information, where the previous set of effective depth information or enriched depth information is determined based on the previous set of second depth information; Obtain feature-level position information based on the image-level pose information, where the feature-level position information is position information after feature alignment of feature points in the second depth information based on feature points in the previous set of valid depth information or enriched depth information; The pose of the mobile device is determined based on the feature-level position information.
11. The device according to any one of claims 7 to 10, characterized in that The processor is further configured to: When the second depth information is key depth information and the second depth information is invalid depth information, a first instruction is sent to the single-line laser radar and / or the dynamic vision sensor, wherein the first instruction is used to instruct to increase the frequency of detection of the physical space where the mobile device is located, and the ratio of the number of key feature points included in the invalid depth information to the total number of feature points is less than a second threshold, and the key feature point is a feature point whose average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
12. The device according to any one of claims 7 to 11, characterized in that The processor is further configured to: When the second depth information is key depth information and the second depth information is rich depth information, a second instruction is sent to the single-line laser radar and / or the dynamic vision sensor, the second instruction being used to instruct to reduce the frequency of detecting the physical space where the mobile device is located, the ratio of the number of key feature points included in the rich depth information to the total number of feature points is greater than a third threshold, and the key feature point is a feature point for which the average value of the difference in pixel values with surrounding feature points is greater than or equal to a fourth threshold.
13. A radar, characterized in that: The radar is used to: After receiving the first instruction, increasing the frequency of detecting the physical space where the radar is located based on the first instruction; and / or, After receiving the second instruction, the frequency of detecting the physical space where the radar is located is reduced based on the second instruction.
14. A dynamic vision sensor, characterized in that: The dynamic vision sensor is used for: After receiving the first instruction, increasing the frequency of detecting the physical space where the dynamic vision sensor is located based on the first instruction; and / or, After receiving the second instruction, the frequency of detecting the physical space where the dynamic vision sensor is located is reduced based on the second instruction.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the computer is caused to perform the method according to any one of claims 1 to 6.
16. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 6.
Citation Information
Cited By
Pose estimation method and related apparatus
EP4715430A1
Pose estimation method and related apparatus
WO2024250726A1
