Event feature point extraction method and device, electronic equipment and storage medium

Abstract

The invention provides an event feature point extraction method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a latest event of a current time window event stream; according to the position of the event pixel point of the newest event, updating the recorded value of the position of the pixel point corresponding to the global event feature map into the event timestamp of the newest event to obtain a newest global event feature map; detecting a grid value of a grid to which the event pixel points belong in a global grid chart which is uniformly divided by the global event feature map; if the grid value of the belonging grid is a first preset value, performing feature point judgment on the event pixel points based on the latest global event feature map; and if the event pixel points are judged to be feature points, updating the grid values of the grids to which the event pixel points belong to a second preset value, and taking the latest global grid chart as a global feature point chart of the current time window. The uniformity of the event feature points can be improved, and the problem that the feature points of the target visual task are easy to lose in the tracking or matching process is reduced.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly relates to an event feature point extraction method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] SAE (Surface of Active Events) is a method for processing the output data of an event camera. By converting the time information of the event stream into spatial information, the event stream can be converted into an image frame, so that traditional computer vision algorithms can be used for processing, and visual tasks such as feature point tracking or matching in Simultaneous Localization and Mapping (SLAM) can be realized using event stream data. However, existing event-based feature point detection methods, such as corner point detection methods, may lead to problems such as uneven feature point distribution and overly dense feature points, resulting in poor feature point uniformity and easy loss during the feature point tracking or matching process. Summary of the Invention

[0003] This application provides an event feature point extraction method, apparatus, electronic device, and computer-readable storage medium, which can improve the uniformity of event feature points and reduce the problem of easy loss during the feature point tracking or matching process of the target visual task.

[0004] In a first aspect, this application provides an event feature point extraction method, and the method includes:

[0005] Obtain the latest event of the current time window event stream of the target visual task;

[0006] According to the position of the event pixel point of the latest event, update the recorded value of the corresponding pixel point position of the global event feature map of the target visual task to the event timestamp of the latest event, and obtain the latest global event feature map;

[0007] In the global grid map evenly divided by the global event feature map, detect the grid value of the grid to which the event pixel point belongs, where each grid in the global grid map corresponds to multiple pixel points of the global event feature map;

[0008] If the grid value of the grid to which the event pixel point belongs is the first preset value, perform feature point determination on the event pixel point based on the latest global event feature map;

[0009] If the event pixel point is determined to be a feature point, update the grid value of the grid to which the event pixel point belongs to the second preset value to obtain the latest global grid, and use the latest global grid map as the global feature point map of the current time window.

[0010] In some embodiments, determining feature points for the event pixel points based on the latest global event feature map includes:

[0011] Segment the latest global event feature map according to a preset local map size and the event pixel points to obtain a local event feature map of the event pixel points;

[0012] Obtain a circular pixel point set of the local event feature map;

[0013] Judge whether the event pixel point is a feature point according to the circular pixel point set.

[0014] In some embodiments, judging whether the event pixel point is a feature point according to the circular pixel point set includes:

[0015] Obtain target pixel points with the latest timestamp from the circular pixel point set;

[0016] Taking the next pixel point of the target pixel point as the starting point of the clockwise pointer and the previous pixel point of the target pixel point as the starting point of the counterclockwise pointer, traverse the circular pixel point set according to a preset traversal rule until the clockwise pointer and the counterclockwise pointer overlap, and obtain the first traversed pixel point length of the clockwise pointer and the second traversed pixel point length of the counterclockwise pointer;

[0017] If both the first traversed pixel point length and the second traversed pixel point length are greater than a preset pixel point length, determine the event pixel point as a feature point.

[0018] In some embodiments, judging whether the event pixel point is a feature point according to the circular pixel point set includes:

[0019] Obtain a pixel point subset with the latest timestamp from the circular pixel point set;

[0020] If there is a continuous pixel point arc in the pixel point subset and the angle of the continuous pixel point arc is within a preset angle range, determine the event pixel point as a feature point.

[0021] In some embodiments, the method further includes:

[0022] When the next time window event stream of the target visual task arrives, perform feature point tracking based on the current time window event stream and the next time window event stream to obtain the grids where the feature points in the global feature point map of the current time window fall after moving;

[0023] In the latest global grid map, update the grid values of the grid cells where each feature point falls after movement and the grid values of the grid cells where each feature point falls before movement.

[0024] In some embodiments, updating the recorded value of the corresponding pixel position in the global event feature map of the target visual task to the event timestamp of the latest event according to the position of the event pixel of the latest event to obtain the latest global event feature map includes:

[0025] When the latest event is a non-redundant event, update the recorded value of the corresponding pixel position in the global event feature map of the target visual task to the event timestamp of the latest event to obtain the latest global event feature map according to the position of the event pixel of the latest event.

[0026] In some embodiments, the method further includes:

[0027] Obtain the recorded value of the corresponding pixel position in the global event feature map of the target visual task according to the position of the event pixel as the comparison timestamp of the latest event; if the comparison timestamp is outside the current time window, determine that the latest event is a non-redundant event;

[0028] Or, if the polarity of the latest event is different from the polarity of the previous event at the event pixel, determine that the latest event is a non-redundant event.

[0029] In some embodiments, the target visual task is at least one of a map construction task of a smart wearable device and a positioning task of a smart wearable device, and the method further includes:

[0030] Perform a map construction task of the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window of the target visual task;

[0031] And / or, perform a positioning task of the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window.

[0032] In a third aspect, the present application further provides an electronic device, which includes a processor and a memory. A computer program is stored in the memory, and when the processor calls the computer program in the memory, it executes any event feature point extraction method provided by the present application.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the event feature point extraction method.

[0034] In this application, in the global grid map evenly divided by the global event feature map, the grid value of the grid to which the event pixel point belongs is detected; if the grid value of the belonging grid is the first preset value, the event pixel point is determined as a feature point based on the latest global event feature map; if the event pixel point is determined as a feature point, the grid value of the belonging grid is updated to the second preset value. In this way, a global grid map can be maintained to record event feature points. On the one hand, since the global grid map is evenly divided based on the global event feature map, the event feature points can be relatively evenly distributed. Since the more evenly the feature points are distributed in space, the more accurately the feature matching can estimate the spatial geometric relationship, it can avoid the problem that feature points cannot be matched due to uneven distribution of feature points, such as being too sparse in some positions, and reduce the problem that the feature point tracking or matching process in the target vision task is prone to loss. On the other hand, since each grid in the global grid map corresponds to multiple pixel points of the global event feature map, and the grid value of each grid is used to record an event feature point, an event feature point can be extracted for the multiple pixel points corresponding to each grid, without extracting an event feature point for each pixel point separately, thus avoiding the problem that the extracted event feature points are too concentrated, improving the extraction speed of event feature points, reducing the redundancy of event feature points, and improving the uniformity of event feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present application;

[0037] Figure 2 is a schematic flowchart of a method for extracting event feature points provided by an embodiment of the present application;

[0038] Figure 3 is an explanatory schematic diagram of the division of the global grid map provided in an embodiment of the present application;

[0039] Figure 4 is a schematic diagram of a scenario for updating the recorded value of the pixel position in the global event feature map provided in an embodiment of the present application;

[0040] Figure 5 is a schematic diagram of a scenario for updating the grid value in the global grid map provided in an embodiment of the present application;

[0041] Figure 6It is a schematic diagram showing an embodiment of the overall process for extracting event feature points provided in an embodiment of the present application;

[0042] Figure 7 It is a schematic structural diagram of an embodiment of an event feature point extraction device provided in an embodiment of the present application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of the present application.

[0044] The flowcharts shown in the accompanying drawings are only illustrative, and do not necessarily include all the content and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to the actual situation.

[0045] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, "a plurality of" means two or more, unless otherwise specifically defined.

[0046] In order to enable any person skilled in the art to implement and use the present application, the following description is given. In the following description, details are set forth for the purpose of explanation. It should be understood that those skilled in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known processes will not be elaborated in detail to avoid unnecessary details from obscuring the description of the embodiments of the present application. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope that conforms to the principles and features disclosed in the embodiments of the present application.

[0047] An embodiment of the present application provides a method and apparatus for extracting event feature points, an electronic device, and a computer-readable storage medium. Among them, the event feature point extraction apparatus can be integrated in the electronic device. Among them, the electronic device can be a mobile robot, smart glasses, a smart helmet, etc. The smart glasses can be AR (augmented reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, XR (eXtended Reality) glasses, etc. The smart helmet can be an AR helmet, etc.

[0048] The execution subject of the event feature point extraction method in the embodiment of the present application can be the event feature point extraction apparatus provided by the embodiment of the present application, or an electronic device integrated with the event feature point extraction apparatus. Among them, the event feature point extraction apparatus can be implemented in a hardware or software manner.

[0049] The following will describe in detail some embodiments of the present application with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0050] Figure 1 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present application.

[0051] As Figure 1 shown, the electronic device 100 includes a processor 101 and a memory 102. The processor 101 and the memory 102 are connected through a bus 103, and this bus is, for example, an I2C (Inter-integrated Circuit) bus.

[0052] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire electronic device 100. The processor 101 can be a central processing unit (CPU), and this processor 101 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or this processor can also be any conventional processor, etc.

[0053] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM), a magnetic disk, an optical disc, a USB flash drive, a mobile hard disk, etc.

[0054] Those skilled in the art can understand that Figure 1 the structure shown in is only a block diagram of some structures related to the solution of the embodiment of the present application, and does not constitute a limitation on the electronic device to which the solution of the embodiment of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0055] Among them, the processor 101 is used to run the computer program stored in the memory 102, and when executing the computer program, implement any one of the event feature point extraction methods provided by the embodiments of the present application. For example, the processor 101 is used to run the computer program stored in the memory 102, and when executing the computer program, the following steps can be implemented:

[0056] Obtain the latest event of the current time window event stream of the target visual task; according to the position of the event pixel point of the latest event, update the recorded value of the corresponding pixel point position of the global event feature map of the target visual task to the event timestamp of the latest event to obtain the latest global event feature map; in the global grid map evenly divided by the global event feature map, detect the grid value of the grid to which the event pixel point belongs, where each grid in the global grid map corresponds to multiple pixel points of the global event feature map; if the grid value of the belonging grid is the first preset value, then perform feature point determination on the event pixel point based on the latest global event feature map; if the event pixel point is determined to be a feature point, then update the grid value of the belonging grid to the second preset value to obtain the latest global grid, and use the latest global grid map as the global feature point map of the current time window.

[0057] It should be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described electronic device can refer to the corresponding process in the embodiment of the following event feature point extraction method, and will not be elaborated here.

[0058] Next, Figure 1 taking the electronic device shown in as the execution subject of the event feature point extraction method as an example, the event feature point extraction method provided by the embodiments of the present application will be introduced in detail. For the sake of simplicity and convenience of description, the execution subject will be omitted in the subsequent method embodiments.

[0059] Please refer to Figure 2 , Figure 2It is a schematic flow chart of an event feature point extraction method provided by an embodiment of the present application. The event feature point extraction method includes steps 201 to 205, where:

[0060] 201. Obtain the latest event of the current time window event stream of the target visual task.

[0061] Among them, the current time window event stream includes multiple first events, and each first event includes the timestamp of the corresponding event and the pixel point position of the corresponding event.

[0062] Among them, the first event is an event detected by the event camera within the current time window.

[0063] Among them, the specific duration value of the time window can be set according to the actual business scenario requirements. For example, if it is necessary to extract a global feature point map every 1 second, then a time window is set every 1 second. Among them, the global feature point map is a feature map used to record the feature points extracted based on the global event feature map.

[0064] Among them, the target visual task is a visual task that requires feature point extraction, and it can be feature point tracking / feature point matching in SLAM; for example, when applying SLAM technology to the navigation of intelligent wearable devices (such as AR glasses), it is feature point tracking / feature point matching in SLAM.

[0065] Exemplarily, each event output by the event camera within the current time window can be received in real time as the latest event. For example, taking the target visual task as feature point tracking / feature point matching in SLAM during AR glasses navigation as an example, within the current time window (such as a period with a duration of 1 s) of the event camera of the AR glasses, each event output by the event camera of the AR glasses in real time can be obtained as the latest event.

[0066] Furthermore, in order to avoid problems such as event redundancy and over-concentration of feature points, it is also possible to determine whether the latest event is a redundant event. When it is determined that the latest event is a redundant event, the latest event is discarded and the next latest event is continued to be obtained; when it is determined that the latest event is a non-redundant event, the process of step 202 is triggered. At this time, step 202 specifically includes: when the latest event is a non-redundant event, according to the position of the event pixel point of the latest event, the recorded value of the corresponding pixel point position of the global event feature map of the target visual task is updated to the event timestamp of the latest event, and the latest global event feature map is obtained. In some embodiments, it is possible to determine whether the latest event is a redundant event based on the following steps A1 to A3, and steps A1 to A3 are as follows:

[0067] A1. Obtain the recorded value of the pixel position corresponding to the global event feature map of the target visual task according to the position of the event pixel of the latest event, and use it as the comparison timestamp of the latest event.

[0068] For example, as Figure 4 shown, assume that the position of the event pixel of the latest event is the pixel position at the a-th row and the b-th column. Then, look up the recorded value of the pixel position at the a-th row and the b-th column in the global event feature map as the comparison timestamp of the latest event. Among them, for the maintenance method of the recorded values of each pixel position in the global event feature map, reference can be made to the relevant description of "global event feature map" later, which will not be elaborated here.

[0069] A2. If the comparison timestamp is outside the current time window, determine that the latest event is a non-redundant event.

[0070] A3. If the comparison timestamp is within the current time window, determine that the latest event is a redundant event.

[0071] In some embodiments, it is possible to determine whether the latest event is a redundant event based on the following steps B1 to B3. Steps B1 to B3 are as follows:

[0072] B1. Detect whether the polarity of the latest event at the event pixel is the same as the polarity of the previous event.

[0073] Among them, the polarity refers to the direction of the brightness change of the event. Exemplarily, it can be divided into positive polarity and negative polarity. For example, when the polarity of the latest event is positive polarity, it means that the brightness change of the event pixel is an increase (i.e., the brightness variable); another example is that when the polarity of the previous event is negative polarity, it means that the brightness change of the pixel corresponding to the previous event is a decrease (i.e., the brightness becomes darker).

[0074] B2. If the polarity of the latest event at the event pixel is different from the polarity of the previous event, determine that the latest event is a non-redundant event.

[0075] B3. If the polarity of the latest event at the event pixel is the same as the polarity of the previous event, determine that the latest event is a redundant event.

[0076] 202. According to the position of the event pixel of the latest event, update the recorded value of the pixel position corresponding to the global event feature map of the target visual task to the event timestamp of the latest event, and obtain the latest global event feature map.

[0077] To better understand the embodiments of the present application, the following first introduces some names involved in the embodiments of the present application:

[0078] 1. Global event feature map: It includes w*h pixel points, and the recorded value at each pixel point position i is used to indicate the timestamp of the latest event at the pixel point position i. The resolution w*h (i.e., size) of the global event feature map is the same as the resolution of the event camera, as shown in Figure 3 (a) in. The difference between the global feature point map and the global event feature map is that: the global event feature map is a feature map used to record the brightness change events of each pixel point, and the global feature point map is a feature map used to record the feature points extracted based on the global event feature map.

[0079] 2. Global grid map: It includes m*n grids, and the grid value of each grid j is used to indicate whether there are event feature points in each grid area. Among them, the grid value of grid j being equal to the first preset value (such as 0) indicates that there are no event feature points in the grid j area, and the grid value of grid j being equal to the second preset value (such as 1) indicates that there are event feature points in the grid j area.

[0080] It can be understood that the specific values of the first preset value and the second preset value here are only examples, and the specific values of the first preset value and the second preset value can be set according to the actual business scenario requirements, and are not limited by the examples here.

[0081] As shown in Figure 3 , the global event feature map with a resolution of w*h is divided into a global grid map containing m*n grids. The final number of event feature points extracted is p, and w, h, m, n, and p are all positive integers, and satisfy the constraint relationship: w / h = m / n and m*n ≥ p.

[0082] In some embodiments, the rectangular area corresponding to the global event feature map can be divided into m*n equal-sized rectangular sub-regions in a uniformly divided manner, and each rectangular sub-region is used as a grid, so as to obtain an m*n global grid map; as shown in Figure 3 , for a global event feature map with w*h = 9*9 (as shown in Figure 3 (a) in), a global grid map with m*n = 3*3 can be divided (as shown in Figure 3 (b) in), and each grid corresponds to multiple pixel points, as shown by the dashed box in Figure 3 (a).

[0083] In some embodiments, a quadtree or other uniformization method can also be used to divide and obtain an m*n global grid map.

[0084] 3. Local event feature map: It is used to reflect the situation around a certain pixel point, and is obtained by dividing the global event feature map according to a preset local map size with a certain pixel point k as the center.

[0085] Among them, an event pixel refers to a pixel where the luminance changes corresponding to the latest event.

[0086] Among them, the position of the event pixel refers to the pixel position corresponding to the latest event, that is, the pixel position where the luminance changes corresponding to the latest event.

[0087] Among them, the event timestamp refers to the timestamp corresponding to the latest event, that is, the timestamp when the luminance changes corresponding to the latest event.

[0088] Specifically, as Figure 4 shown, Figure 4 FIG. is a schematic diagram of a scenario for updating the recorded value of the pixel position in the global event feature map provided in the embodiment of the present application. When the latest event is detected, according to the position of the event pixel of the latest event, the recorded value of the corresponding pixel position in the global event feature map is updated to the event timestamp of the latest event. For example, if the timestamp corresponding to the latest event is 12:10:30 and the pixel position corresponding to the latest event is at the a-th row and b-th column, then the recorded value at the a-th row and b-th column in the global event feature map is updated to: 12:10:30.

[0089] 203. In the global grid map evenly divided by the global event feature map, detect the grid value of the grid to which the event pixel belongs.

[0090] Among them, each grid in the global grid map corresponds to multiple pixels in the global event feature map.

[0091] Among them, the belonging grid refers to the grid in each grid of the global grid map into which the event pixel falls.

[0092] Please refer to Figure 5 , Figure 5 FIG. is a schematic diagram of a scenario for updating the grid value in the global grid map provided in the embodiment of the present application. For example, assume that the event pixel A (as Figure 5 shown in (a) of FIG., Figure 5 the pixels within the dashed box 1 in FIG. are all correspondingly divided into grid 1) is divided into grid 1 (as Figure 5 shown in (b) of FIG.), then the belonging grid of the event pixel A is "grid 1"; at this time, the grid value of the belonging grid (such as "grid 1") can be detected in the global grid map (as Figure 5 shown in (c) of FIG., and the grid value of the belonging grid of the detected event pixel A is "0" at this time).

[0093] When it is detected in step 203 that the grid value of the grid to which the event pixel belongs is the first preset value, it proves that there is no event feature point in the grid to which the event pixel belongs. At this time, step 204 is entered to determine whether the event pixel is a feature point, so as to update the grid value of the grid to which the event pixel belongs. When it is detected in step 203 that the grid value of the grid to which the event pixel belongs is the second preset value, it proves that there is already an event feature point in the grid to which the event pixel belongs. At this time, the determination of the feature point for the event pixel is skipped, and wait for the arrival of the next latest event to reduce the redundancy of the extraction of event feature points. It can be seen that, on the one hand, by maintaining the global grid map to record event feature points, the problem that the extracted event feature points are too concentrated can be avoided, the redundancy of event feature points can be reduced, and thus the uniformity of event feature points can be improved. Since the more evenly the feature points are distributed in space, the more accurately the feature matching can estimate the spatial geometric relationship, the problem that feature point pairs cannot be matched due to uneven distribution of feature points (such as the event data collected before and after the camera view moves at relatively sparse feature point distributions cannot capture matching feature point pairs) can be avoided. Therefore, by maintaining the global grid map to record event feature points, the problem that the feature point tracking or matching process in the target vision task is prone to loss can be reduced, and the pose estimation accuracy of the target vision task can be improved.

[0094] On the other hand, by maintaining a homogenized global grid map to record event feature points, since each grid in the global grid map corresponds to multiple pixel points and the grid value of each grid is used to record an event feature point, an event feature point can be extracted for the multiple pixel points corresponding to each grid, without extracting an event feature point for each pixel point respectively. Thus, the problem that the extracted event feature points are too concentrated can be avoided, the extraction speed of event feature points can be increased, the redundancy of event feature points can be reduced, and the uniformity of event feature points can be improved.

[0095] 204. If the grid value of the grid to which it belongs is the first preset value, then perform feature point determination on the event pixel based on the latest global event feature map.

[0096] There are multiple implementation manners for step 204. Exemplarily, they include:

[0097] (1) In some embodiments, it is possible to determine whether the event pixel is a feature point in the following manners of steps 2041A to 2046A:

[0098] 2041A. Segment the latest global event feature map according to the preset local map size and the event pixel to obtain the local event feature map of the event pixel.

[0099] Among them, the specific value of the preset local map size can be set according to the requirements of the actual business scenario. In this embodiment, the specific value of the preset local map size is not limited. For example, the preset local map size can be 9*9.

[0100] Exemplarily, as Figure 5 shown, taking the event pixel point (such as event pixel point A) as the center point, and segmenting the latest global event feature map (such as the resolution of the latest global event feature map is 1280*720) according to the preset local map size (such as 3*3), the local event feature map of the event pixel point (such as event pixel point A) can be obtained. For example, the area shown by the dashed box 2 in Figure 5 (a) is segmented to obtain the local map shown in Figure 5 (d) as the local event feature map of event pixel point A.

[0101] 2042A. Obtain the circular pixel point set of the local event feature map.

[0102] For example, as Figure 5 shown in Figure 5 (d), a circle can be selected from the local event feature map according to the preset radius (such as a radius of 3 pixel points or a radius of 4 pixel points). The set of pixel points included in the circle with the preset radius is the circular pixel point set. For example, when a circle with a radius of 3 is taken, the circular pixel point set is the set of pixel points included in the solid circle in Figure 5 (d). Another example, when a circle with a radius of 4 is taken, the circular pixel point set is the set of pixel points included in the dashed circle in

[0103] 2043A. From the circular pixel point set, obtain the target pixel point whose corresponding timestamp is the latest timestamp.

[0104] Among them, the target pixel point refers to the pixel point corresponding to the latest timestamp among all the timestamps recorded by the circular pixel point set.

[0105] For example, as Figure 5 shown in Figure 5 (d), assuming that the circular pixel point set is the set of pixel points included in the dashed circle in

[0106] (d), and the timestamp recorded by pixel point B is the latest timestamp in the circular pixel point set, then pixel point B is taken as the target pixel point. Starting from the next pixel point of the target pixel point as the starting point of the clockwise pointer and the previous pixel point of the target pixel point as the starting point of the counterclockwise pointer, traverse the circular pixel point set according to the preset traversal rule until the clockwise pointer and the counterclockwise pointer overlap, and obtain the first traversed pixel point length of the clockwise pointer and the second traversed pixel point length of the counterclockwise pointer.

[0107] Among them, the length of the first traversed pixel points refers to the length of the pixel points traversed by the clockwise pointer when the clockwise pointer and the counterclockwise pointer overlap.

[0108] Among them, the length of the second traversed pixel points refers to the length of the pixel points traversed by the counterclockwise pointer when the clockwise pointer and the counterclockwise pointer overlap.

[0109] For ease of understanding, continue with the example of step 2043A. For example, as Figure 5 shown in (d), assume that the circular pixel point set is Figure 5 the set of pixel points included in the dashed circle in (d). Taking the next pixel point of the target pixel point (such as pixel point C) as the starting point of the clockwise pointer and the previous pixel point of the target pixel point (such as pixel point D) as the starting point of the counterclockwise pointer, traverse the circular pixel point set according to a preset traversal rule (for example, if the timestamp recorded at the pixel point position where the clockwise pointer is located is newer than the timestamp recorded at the pixel point position where the counterclockwise pointer is located, then the clockwise pointer moves clockwise from its current pixel point to the next pixel point in the circular pixel point set; otherwise, if the timestamp recorded at the pixel point position where the counterclockwise pointer is located is newer than the timestamp recorded at the pixel point position where the clockwise pointer is located, then the counterclockwise pointer moves counterclockwise from its current pixel point to the next pixel point in the circular pixel point set) until the clockwise pointer and the counterclockwise pointer overlap (such as overlapping at pixel point F), and obtain the length of the first traversed pixel points of the clockwise pointer (for example, when overlapping at pixel point F, the length of the first traversed pixel points is 6 pixel point lengths), and the length of the second traversed pixel points of the counterclockwise pointer (for example, when overlapping at pixel point F, the length of the second traversed pixel points is 10 pixel point lengths).

[0110] 2045A. If both the length of the first traversed pixel points and the length of the second traversed pixel points are greater than a preset pixel point length, then determine the event pixel point as a feature point.

[0111] For ease of understanding, continue with the example of step 2044A. For example, as Figure 5 shown in (d), if both the length of the first traversed pixel points (such as 6 pixel point lengths) and the length of the second traversed pixel points (such as 10 pixel point lengths) are greater than a preset pixel point length (such as 4 pixel point lengths), then determine the event pixel point (such as event pixel point A) as a feature point.

[0112] 2046A. If at least one of the length of the first traversed pixel points and the length of the second traversed pixel points is less than or equal to the preset pixel point length, then determine that the event pixel point is not a feature point.

[0113] For example, if the length of the first traversed pixel points (such as 13 pixel point lengths) is greater than the preset pixel point length (such as 4 pixel point lengths), but the length of the second traversed pixel points (such as 3 pixel point lengths) is less than the preset pixel point length (such as 4 pixel point lengths), then it is determined that the event pixel point (such as event pixel point A) is not a feature point.

[0114] For another example, if the length of the second traversed pixel points (such as 12 pixel point lengths) is greater than the preset pixel point length (such as 4 pixel point lengths), but the length of the first traversed pixel points (such as 4 pixel point lengths) is equal to the preset pixel point length (such as 4 pixel point lengths), then it is determined that the event pixel point (such as event pixel point A) is not a feature point.

[0115] (2) In some embodiments, the event pixel point can be determined whether it is a feature point through the following steps 2041B to 2046B:

[0116] 2041B. Segment the latest global event feature map according to the preset local map size and the event pixel point to obtain the local event feature map of the event pixel point.

[0117] 2042B. Obtain the circular pixel point set of the local event feature map.

[0118] Steps 2041B to 2042B are similar to steps 2041A to 2042A. For specific details, reference can be made to the relevant descriptions above, and details will not be repeated here.

[0119] 2043B. Obtain the pixel point subset corresponding to the latest timestamp from the circular pixel point set.

[0120] For example, as Figure 5 shown in (d) below, assuming that the circular pixel point set is Figure 5 the set of pixel points included in the solid circle in (d) below. If the timestamps recorded by pixel points 1, 2, 3, 4, and 5 in the circular pixel point set are all the latest timestamps, then the set of pixel points 1, 2, 3, 4, and 5 can be used as the pixel point subset.

[0121] 2044B. If there is a continuous pixel point arc in the pixel point subset and the angle of the continuous pixel point arc is within the preset angle range, then determine the event pixel point as a feature point.

[0122] Among them, the value range of the preset angle range can be set according to the actual scenario requirements. The specific value of the preset angle range is not limited here. For example, the preset angle range can be greater than 30°.

[0123] Among them, the existence of a continuous pixel point arc in the pixel point subset means that there are a continuous plurality of pixel points in the pixel point subset that form an arc shape.

[0124] For ease of understanding, continue the description with the example in step 2043B. For example, as shown in (d) below, if there is a continuous pixel arc in the pixel subset, and the angle of the continuous pixel arc (as shown by the θ angle in (d)) is within a preset angle range, then the event pixel (such as event pixel A) is determined as a feature point. Figure 5 as shown in (d) below, if there is a continuous pixel arc in the pixel subset, and the angle of the continuous pixel arc (as shown by the θ angle in (d)) is within a preset angle range, then the event pixel (such as event pixel A) is determined as a feature point. Figure 5 (as shown by the θ angle in (d)) is within a preset angle range, then the event pixel (such as event pixel A) is determined as a feature point.

[0125] 2045B. If there is no continuous pixel arc in the pixel subset and the angle of the continuous pixel arc is within a preset angle range, then it is determined that the event pixel is not a feature point.

[0126] 2046B. If there is a continuous pixel arc in the pixel subset and the angle of the continuous pixel arc is outside the preset angle range, then it is determined that the event pixel is not a feature point.

[0127] 205. If the event pixel is determined as a feature point, then update the grid value of the corresponding grid to a second preset value to obtain the latest global grid, and use the latest global grid map as the global feature point map of the current time window.

[0128] For example, as shown in Figure 5 below, if the event pixel (such as event pixel A) is determined as a feature point, then update the grid value of the grid corresponding to the event pixel (such as event pixel A) from the first preset value (such as 0) to the second preset value (such as 1), to indicate that there is an event feature point in the grid corresponding to the event pixel (such as event pixel A).

[0129] After all events in the event stream of the current time window are processed to obtain the latest global grid, the latest global grid map can be used as the global feature point map of the current time window, so that the feature points of the target visual task in the current time window can be obtained, which is convenient for subsequent target visual tasks.

[0130] Further, the method further includes: when the event stream of the next time window of the target visual task arrives, performing feature point tracking based on the current time window event stream and the next time window event stream to obtain the grids into which the feature points in the global feature point map of the current time window fall after moving; in the latest global grid map, updating the grid values of the grids into which the feature points fall after moving and the grid values of the grids into which the feature points fall before moving. For example, an optical flow tracking algorithm or other tracking algorithms can be used to detect the image positions of each feature point in the global feature point map of the current time window in two consecutive frames constructed based on the event stream (such as the image frames constructed by the current time window event stream and the next time window event stream), so as to determine the grids into which the feature points in the global feature point map of the previous time window fall after moving. At this time, in the latest global grid map, the grid values of the grids into which the feature points fall after moving can be updated to a second preset value, and the grid values of the grids into which the feature points fall before moving are updated from the second preset value to a first preset value. In this way, the feature point information can be continuously used to facilitate pose estimation.

[0131] Further, when the global feature point map of the current time window is obtained, the pose of the device (such as an AR glasses) can also be estimated by using the global feature point map of the current time window and the global feature point map of the next time window, so as to perform a map construction task or a positioning task according to the pose of the device (such as an AR glasses). For example, in some embodiments, the target visual task can be a map construction task of a smart wearable device. At this time, the method further includes: performing a map construction task of the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window of the target visual task.

[0132] For another example, in some embodiments, the target visual task is a positioning task of a smart wearable device. At this time, the method further includes: performing a positioning task of the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window.

[0133] As one of the embodiments, as Figure 6 shown, the overall process of event feature point extraction can be as follows:

[0134] 1. When the event stream arrives, perform redundant event filtering processing on the latest event. (As shown in the "Event Filtering" part in Figure 6 )

[0135] 2. If the latest event is not a redundant event, the recorded value of the event timestamp of the latest event for the corresponding pixel position in the global event feature map can be updated. (As shown in the "Updating the Global Event Feature Map" part in Figure 6 )

[0136] 3. Detect the grid value of the grid to which the event pixel belongs. (As shown in the "Detect Event Grid Value" section of Figure 6 ).

[0137] 4. If the grid value of the grid to which it belongs is the first preset value (such as 0), then segment the local event feature map of the event pixel (as shown in the "Extract Local Event Feature Map" section of Figure 6 ). If the grid value of the grid to which it belongs is the second preset value (such as 1), then end the processing of the latest event to skip the latest event and wait for the next latest event in the event stream.

[0138] 5. According to the local event feature map of the event pixel, perform feature point determination on the event pixel. (As shown in the "Filter Feature Points" section of Figure 6 ).

[0139] 6. If the event pixel is determined to be a feature point, then update the grid value of the grid to which it belongs in the global grid map to the second preset value, and end the processing of the latest event. (As shown in the "Update Global Grid Map" section of Figure 6 ).

[0140] As can be seen from the above, by detecting the grid value of the grid to which the event pixel belongs in the global grid map evenly divided by the global event feature map; if the grid value of the grid to which it belongs is the first preset value, then perform feature point determination on the event pixel based on the latest global event feature map; if the event pixel is determined to be a feature point, then update the grid value of the grid to which it belongs to the second preset value. In this way, a global grid map can be maintained to record event feature points. On the one hand, since the global grid map is evenly divided based on the global event feature map, the event feature points can be relatively evenly distributed, reducing the problem of easy loss in the feature point tracking or matching process of the target vision task; on the other hand, since each grid in the global grid map corresponds to multiple pixels of the global event feature map, and the grid value of each grid is used to record an event feature point, an event feature point can be extracted for the multiple pixels corresponding to each grid, without extracting an event feature point for each pixel separately, thus avoiding the problem of overly concentrated extracted event feature points, improving the extraction speed of event feature points, reducing the redundancy of event feature points, improving the uniformity of event feature points, and reducing the problem of easy loss in the feature point tracking or matching process of the target vision task.

[0141] In addition, in order to better implement the event feature point extraction method in the embodiments of the present application, on the basis of the event feature point extraction method, an event feature point extraction device is also provided in the embodiments of the present application, as shown in Figure 7 which is a schematic structural diagram of an embodiment of the event feature point extraction device provided by the embodiments of the present application. The event feature point extraction device 700 includes:

[0142] An acquisition unit 701, configured to acquire the latest event of the current time window event stream of the target vision task;

[0143] An update unit 702, configured to update the recorded value of the corresponding pixel position of the global event feature map of the target vision task to the event timestamp of the latest event according to the position of the event pixel of the latest event, so as to obtain the latest global event feature map;

[0144] A detection unit 703, configured to detect the grid value of the grid to which the event pixel belongs in the global grid map evenly divided by the global event feature map, where each grid in the global grid map corresponds to multiple pixels of the global event feature map;

[0145] An extraction unit 704, configured to perform feature point determination on the event pixel based on the latest global event feature map if the grid value of the grid to which the event pixel belongs is a first preset value;

[0146] The extraction unit 704 is further configured to update the grid value of the grid to which the event pixel belongs to a second preset value to obtain the latest global grid if the event pixel is determined to be a feature point, and use the latest global grid map as the global feature point map of the current time window.

[0147] In some embodiments, the extraction unit 704 is configured to:

[0148] Segment the latest global event feature map according to a preset local map size and the event pixel to obtain a local event feature map of the event pixel;

[0149] Obtain a circular pixel point set of the local event feature map;

[0150] Judge whether the event pixel is a feature point according to the circular pixel point set.

[0151] In some embodiments, the extraction unit 704 is configured to:

[0152] Obtain a target pixel point with the latest timestamp from the circular pixel point set;

[0153] Taking the next pixel point of the target pixel point as the starting point of the clockwise pointer and the previous pixel point of the target pixel point as the starting point of the counterclockwise pointer, traverse the circular pixel point set according to a preset traversal rule until the clockwise pointer and the counterclockwise pointer overlap, and obtain the first traversed pixel point length of the clockwise pointer and the second traversed pixel point length of the counterclockwise pointer;

[0154] If both the length of the first traversed pixel points and the length of the second traversed pixel points are greater than a preset pixel point length, then determine the event pixel points as feature points.

[0155] In some embodiments, the extraction unit 704 is configured to:

[0156] Obtain a subset of pixel points with the latest timestamp from the circular pixel point set;

[0157] If there is a continuous pixel point arc in the subset of pixel points and the angle of the continuous pixel point arc is within a preset angle range, then determine the event pixel points as feature points.

[0158] In some embodiments, the updating unit 702 is configured to:

[0159] When the event stream of the next time window of the target visual task arrives, perform feature point tracking based on the current time window event stream and the event stream of the next time window to obtain the grids where the feature points move to in the global feature point map of the current time window;

[0160] In the latest global grid map, update the grid values of the grids where the feature points move to and the grid values of the grids where the feature points move from.

[0161] In some embodiments, the updating unit 702 is configured to:

[0162] When the latest event is a non-redundant event, update the recorded value at the corresponding pixel position in the global event feature map of the target visual task to the event timestamp of the latest event according to the position of the event pixel points of the latest event, to obtain the latest global event feature map;

[0163] The method further includes:

[0164] Obtain the recorded value at the corresponding pixel position in the global event feature map of the target visual task according to the position of the event pixel points of the latest event, as the comparison timestamp of the latest event; if the comparison timestamp is outside the current time window, then determine the latest event as a non-redundant event;

[0165] Alternatively, if the polarity of the latest event is different from the polarity of the previous event at the event pixel points, determine the latest event as a non-redundant event.

[0166] In some embodiments, the event feature point extraction device further includes an execution unit (not shown in the figure), and the execution unit is configured to:

[0167] Perform a map construction task for the intelligent wearable device based on the global feature point map of the current time window and the global feature point map of the next time window of the target visual task;

[0168] And / or, perform a positioning task for the intelligent wearable device based on the global feature point map of the current time window and the global feature point map of the next time window.

[0169] In specific implementation, each of the above units can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above units, reference can be made to the embodiments of the event feature point extraction method described above, which will not be elaborated here.

[0170] Those of ordinary skill in the art can understand that all or part of the steps in the above event feature point extraction method can be completed by instructions, or by controlling relevant hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0171] Therefore, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and these computer programs can be loaded by a processor to execute any event feature point extraction method provided by the embodiments of the present application.

[0172] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0173] In the above embodiments of the event feature point extraction device, computer-readable storage medium, and electronic device, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes and the beneficial effects that can be brought by the above-described event feature point extraction device, computer-readable storage medium, electronic device and their corresponding units can refer to the description of the event feature point extraction method in the above embodiments, which will not be elaborated here specifically.

[0174] The above has introduced in detail an event feature point extraction method, device, electronic device, and computer-readable storage medium provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for extracting event feature points, characterized in that: The method comprises: Get the latest event of the current time window event stream of the target visual task; According to the position of the event pixel of the latest event, the recorded value of the pixel position corresponding to the global event feature map of the target visual task is updated to the event timestamp of the latest event to obtain the latest global event feature map; In a global grid map evenly divided by the global event feature map, detecting a grid value of a grid to which the event pixel point belongs, wherein each grid in the global grid map corresponds to a plurality of pixel points of the global event feature map; If the grid value of the grid to which the event belongs is a first preset value, then feature point determination is performed on the event pixel point based on the latest global event feature map; If the event pixel point is determined to be a feature point, the grid value of the grid to which it belongs is updated to a second preset value to obtain the latest global grid, and the latest global grid map is used as the global feature point map of the current time window.

2. The event feature point extraction method according to claim 1, characterized in that: The determining of feature points of the event pixels based on the latest global event feature map includes: Segmenting the latest global event feature map according to a preset local map size and the event pixel points to obtain a local event feature map of the event pixel points; Obtaining a circular pixel point set of the local event feature map; According to the circular pixel point set, it is determined whether the event pixel point is a feature point.

3. The event feature point extraction method according to claim 2, characterized in that: The step of judging whether the event pixel point is a feature point according to the circular pixel point set includes: From the circular pixel point set, obtain a target pixel point whose corresponding timestamp is the latest timestamp; Taking the next pixel point of the target pixel point as the starting point of the clockwise pointer and the previous pixel point of the target pixel point as the starting point of the counterclockwise pointer, traversing the circular pixel point set according to a preset traversal rule until the clockwise pointer and the counterclockwise pointer overlap, obtaining a first traversal pixel point length of the clockwise pointer and a second traversal pixel point length of the counterclockwise pointer; If the first traversal pixel point length and the second traversal pixel point length are both greater than a preset pixel point length, the event pixel point is determined as a feature point.

4. The event feature point extraction method according to claim 2, characterized in that: The step of judging whether the event pixel point is a feature point according to the circular pixel point set includes: From the circular pixel point set, obtain a pixel point subset whose corresponding timestamp is the latest timestamp; If there is a continuous pixel point arc in the pixel point subset, and the angle of the continuous pixel point arc is within a preset angle range, the event pixel point is determined as a feature point.

5. The event feature point extraction method according to claim 1, characterized in that: The method further comprises: When the next time window event stream of the target visual task arrives, feature point tracking is performed based on the current time window event stream and the next time window event stream to obtain a grid where each feature point in the global feature point map of the current time window falls after movement; In the latest global grid map, the grid values ​​of the feature points after they are moved and the grid values ​​of the feature points before they are moved are updated.

6. The event feature point extraction method according to claim 1, characterized in that: The method of updating the recorded value of the pixel position corresponding to the global event feature map of the target visual task to the event timestamp of the latest event according to the position of the event pixel of the latest event to obtain the latest global event feature map includes: When the latest event is a non-redundant event, according to the position of the event pixel of the latest event, the record value of the pixel position corresponding to the global event feature map of the target visual task is updated to the event timestamp of the latest event to obtain the latest global event feature map; The method further comprises: According to the position of the event pixel point, obtain the record value of the pixel point position corresponding to the global event feature map of the target visual task as the comparison timestamp of the latest event; if the comparison timestamp is outside the current time window, determine that the latest event is a non-redundant event; Alternatively, if the polarity of the latest event is different from the polarity of the previous event at the event pixel point, the latest event is determined to be a non-redundant event.

7. The event feature point extraction method according to claim 1, characterized in that: The target visual task is at least one of a map construction task of a smart wearable device and a positioning task of a smart wearable device, and the method further includes: Perform a map construction task for the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window of the target visual task; And / or, performing a positioning task of the smart wearable device according to the global feature point map of the current time window and the global feature point map of the next time window.

8. An event feature point extraction device, characterized in that: The event feature point extraction device comprises: An acquisition unit, used to acquire the latest event of the current time window event stream of the target visual task; An updating unit, configured to update the recorded value of the pixel position corresponding to the global event feature map of the target visual task to the event timestamp of the latest event according to the position of the event pixel of the latest event, so as to obtain the latest global event feature map; A detection unit, configured to detect a grid value of a grid to which the event pixel point belongs in a global grid map evenly divided by the global event feature map, wherein each grid in the global grid map corresponds to a plurality of pixel points of the global event feature map; An extraction unit, configured to perform feature point determination on the event pixel point based on the latest global event feature map if the grid value of the grid to which the event belongs is a first preset value; The extraction unit is also used to update the grid value of the grid to which it belongs to a second preset value to obtain the latest global grid if the event pixel point is determined to be a feature point, and use the latest global grid map as the global feature point map of the current time window.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method for extracting event feature points according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the event feature point extraction method according to any one of claims 1 to 7.

Images