Visual SLAM front-end optimization method and device for complex scene
By optimizing feature detection and brightness variation in event data, the stability problem of visual SLAM in complex scenes is solved, achieving efficient feature extraction and tracking, and improving the accuracy and robustness of visual SLAM.
Patent Information
- Application Number
- CN202511062541.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-18
AI Technical Summary
Existing visual SLAM front-ends based on event cameras using feature points struggle to construct stable visual representations in complex scenarios such as high dynamic range or low illumination, and also suffer from spatiotemporal error issues.
By acquiring event data for feature detection, extracting event corner feature flows, generating actual and predicted brightness changes, constructing an objective function and minimizing photometric residuals, optimizing motion parameters and optical flow, and achieving stable feature tracking.
Stable feature extraction and tracking were achieved in complex scenarios, reducing computational load and improving the accuracy and robustness of visual SLAM.
Smart Images
Figure CN120976262A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of visual SLAM technology, and particularly relates to a visual SLAM front-end optimization method and device for a complex scene, a storage medium and an electronic device. BACKGROUND
[0002] The visual SLAM (Simultaneous Localization and Mapping) front-end mainly estimates the rough camera motion according to the information of adjacent images, and can be divided into a feature point method and a direct method according to different calculations. The visual SLAM front-end of the feature point method constructs an accurate descriptor for features, and finds similar descriptors in adjacent images for pairing. The feature point method can achieve high accuracy under the condition of stable visual information, and occupies a dominant position in the SLAM method.
[0003] At present, the feature point method visual SLAM front-end based on an event camera mostly constructs a frame image through event data, and adopts a traditional image method for feature extraction and tracking. Although this method effectively solves the problem of inconsistent modalities, there is a problem that it cannot adapt to complex scenes such as high dynamics or low light due to the large noise of event data on the existing feature extraction and tracking algorithm. It is difficult to construct a stable visual expression using a few event images, and a large number of event images will produce a motion trajectory, causing a space-time error. SUMMARY
[0004] The embodiments of the present application provide a visual SLAM front-end optimization method and device for a complex scene, a storage medium and an electronic device, which can adapt to complex environments and realize stable feature point extraction and tracking.
[0005] The embodiments of the present application provide a visual SLAM front-end optimization method for a complex scene, comprising: Obtaining a scene image and event data of a corresponding scene through an event camera; Performing feature detection based on the event data to obtain an event corner feature flow; Extracting the time corner feature flow, generating an actual brightness change amount, generating a predicted brightness change amount based on the scene image, constructing a target function based on the actual brightness change amount and the predicted brightness change amount, minimizing the target function, and obtaining optimized motion parameters and an optical flow.
[0006] Further, according to the visual SLAM front-end optimization method for a complex scene, wherein performing feature detection based on the event data to obtain an event corner feature flow comprises: Filtering events in the event data and constructing a G-SAE; extracting an event-centered local SAE by G-SAE to generate a local patch, calculating a candidate feature point score using the local patch; determining the candidate feature point as an event feature point when the candidate feature point score is greater than a preset score threshold; constructing a feature gray image based on a feature value of the event feature point, and performing a dilation operation and a non-maximum suppression operation to obtain an event feature point with a maximum feature value as a coarse corner point; creating a mask image with a gray value of 255, sorting and traversing the coarse corner points according to the feature values, and selecting a point with a feature value of 255 as a fine feature point, a set of the fine feature points being an event corner feature stream.
[0007] Further, according to the above visual SLAM front-end optimization method for complex scenes, wherein the candidate feature point score is calculated by the following formula:
[0008] wherein, is the candidate feature point score, is a determinant, is a trace of a matrix, is an eigenmatrix, is an eigenvalue of the eigenmatrix M, is a matrix element of the eigenmatrix M, is a constant;
[0009]
[0010]
[0011]
[0012] wherein, are gradients of the local patch in x and y directions respectively, is a Gaussian weighting function.
[0013] Further, according to the above visual SLAM front-end optimization method for complex scenes, wherein the event corner feature stream is extracted to generate an actual brightness change amount, including: when an event in the event corner feature stream is triggered, at time , once the brightness of the event on the pixel increases by more than a set threshold ±C, the brightness increase amount at this time is:
[0014] wherein, t is the time of the last event on the same pixel, ∈{-1, +1} is the event polarity, is the brightness increase, is the time of the brightness signal, is the brightness signal at time ; calculating an actual brightness change based on the brightness increase:
[0015] wherein, is the actual brightness change, is a discrete parameter.
[0016] Further, according to the above visual SLAM front-end optimization method for complex scenes, wherein the predicted brightness change is generated based on the scene image, comprising: calculating a gradient map of the scene image, and calculating a predicted brightness change based on the current motion parameter and the gradient:
[0017] wherein, is the predicted brightness change, is a warped image not used for registration, is the current motion parameter, is the gradient, is the elapsed time.
[0018] Further, according to the above visual SLAM front-end optimization method for complex scenes, wherein the target function is constructed based on the actual brightness change and the predicted brightness change, comprising: differencing the actual brightness change and the predicted brightness change to obtain a photometric residual, and taking the photometric residual as the target function.
[0019] Further, according to the above visual SLAM front-end optimization method for complex scenes, wherein the target function is minimized to obtain the optimized motion parameter and the optical flow, comprising: minimizing the norm of the photometric residual:
[0020] proposing to minimize the difference between the unit norm patches:
[0021] considering the warping caused by the rigid body motion on the image plane:
[0022] wherein R(P) is a two-dimensional rotation matrix, t(p) is a translation vector; The optimal motion parameter p and the optical flow v are selected to minimize the target function, so as to obtain the optimized motion parameter and the optical flow.
[0023] The embodiment of the present application also provides a visual SLAM front-end optimization device for a complex scene, comprising: An acquisition module is configured to acquire a scene image and acquire event data of a corresponding scene through an event camera; A feature extraction module is configured to perform feature detection based on the event data to obtain an event corner feature flow; An optimization module is configured to extract the time corner feature flow, generate an actual brightness change amount, generate a predicted brightness change amount based on the scene image, construct a target function based on the actual brightness change amount and the predicted brightness change amount, minimize the target function, and obtain an optimized motion parameter and an optical flow.
[0024] The embodiment of the present application also provides a computer readable storage medium, wherein a plurality of instructions are stored in the computer readable storage medium, and the instructions are suitable for being loaded by a processor to execute any one of the visual SLAM front-end optimization methods for a complex scene.
[0025] The embodiment of the present application also provides an electronic device, comprising a processor and a memory, wherein the processor is electrically connected with the memory, the memory is configured to store instructions and data, and the processor is configured to execute steps in any one of the visual SLAM front-end optimization methods for a complex scene.
[0026] The visual SLAM front-end optimization method, device, storage medium and electronic device provided by the present application perform feature detection by using event data, construct a target function based on an actual brightness change amount and a predicted brightness change amount, optimize a motion parameter and an optical flow, realize feature rootness in a complex scene, and effectively reduce the amount of calculation. BRIEF DESCRIPTION OF DRAWINGS
[0027] The technical solutions and other beneficial effects of the present application will become apparent through the following detailed description of the specific embodiments of the present application in combination with the accompanying drawings.
[0028] Figure 1 A flowchart of the visual SLAM front-end optimization method for a complex scene provided by the embodiment of the present application.
[0029] Figure 2 A structure schematic diagram of the visual SLAM front-end optimization device for a complex scene provided by the embodiment of the present application.
[0030] Figure 3A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0032] An embodiment of the present application provides a visual SLAM front-end optimization method for a complex scene, a device, a storage medium and an electronic device. The device for the visual SLAM front-end optimization for the complex scene provided by an embodiment of the present application can be integrated in an electronic device, which can be a terminal, a server or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box or other devices.
[0033] Please refer to Figure 1 , Figure 1 A flowchart of the visual SLAM front-end optimization method for the complex scene provided by an embodiment of the present application is applied to an electronic device, and the visual SLAM front-end optimization method for the complex scene includes the following steps. S1, acquiring a scene image and acquiring event data of a corresponding scene through an event camera.
[0034] S2, performing feature detection based on the event data to obtain an event corner feature flow.
[0035] The feature points are extracted by using a Modified FA-Harris feature detection algorithm improved from an asynchronous Fa-Harris algorithm. Specifically, the following steps are included. S21, filtering events in the event data and constructing a G-SAE.
[0036] An event filter eFilter is applied, and only when a time of a previous event exceeds a fixed window (i.e., a preset time threshold) or a polarity of the previous event is different from a polarity of an input event, the filtered event is updated (an update mode is that valid events are selected to replace the original event stream through a time window and a polarity change condition). S22, extracting a local SAE centered on an event through the G-SAE to generate a local patch, and using the local patch to calculate a candidate feature point score.
[0037] The filtered events are used for the construction and update of G-SAE (Gradient-Sensitive Asynchronous Event Representation, which is essentially an improved version of traditional SAE, which introduces a gradient weight factor on the basis of SAE, and emphasizes the importance of events in high gradient areas more. The 9x9 event-centered local SAE is extracted through G-SAE, and the nearest N events (N=25) are found on the local SAE (Surface of Active Events) to generate a local patch, and the candidate feature point score is calculated using the local patch as follows:
[0038] wherein, is the candidate feature point score, is the determinant, is the trace of the matrix, is the eigenmatrix, is the eigenvalue of the eigenmatrix M, is the matrix element of the eigenmatrix M, is a constant, ;
[0039]
[0040]
[0041]
[0042] wherein, are the gradients of the local patch in the x and y directions, respectively, is a Gaussian weighting function. Specifically, a Gaussian weighting function with pixel expansion is as follows:
[0043] S23, when the candidate feature point score is greater than the preset score threshold, the candidate feature point is determined as an event feature point.
[0044] S24, a feature gray image is constructed based on the eigenvalue of the event feature point, and a dilation operation and a non-maximum suppression are performed to obtain an event feature point with the maximum eigenvalue, which is taken as a coarse corner point.
[0045] S25, a mask image with a gray value of 255 is created, the coarse corner points are sorted and traversed according to the eigenvalues, and a point with an eigenvalue of 255 is selected as a fine feature point. The set of fine feature points is an event corner feature stream.
[0046] The score obtained by calculation is compared with a threshold value, and when the score of the candidate feature point is greater than a preset score threshold value, the event feature point candidate is determined as an event feature point. Then a feature gray scale image is constructed with the feature point and the feature value (the eigenvalue of the feature matrix M), and a 3*3 kernel inflation operation and non-maximum suppression are performed to obtain the event feature point with the maximum feature value, which is recorded as a "coarse corner point". Finally, a mask image with a gray value of 255 is created, the coarse corner points are sorted according to the feature values (from large to small or from small to large) and traversed, and the point with a feature value of 255 is selected as a fine feature point, so that fine screening and extraction of the feature point are realized.
[0047] S3, extracting the time corner point feature flow, generating an actual brightness change amount, generating a predicted brightness change amount based on the scene image, constructing a target function based on the actual brightness change amount and the predicted brightness change amount, minimizing the target function, and obtaining the optimized motion parameter and the optical flow.
[0048] In an embodiment, step S3 comprises: An event camera can respond to changes in a continuous brightness signal at each pixel. When an event in the time corner point feature flow is triggered, at time , once the brightness increase of the event at the pixel exceeds a set threshold value ±C (C>0), the brightness increase amount at this time is: (1) wherein, is the time of the previous event at the same pixel, is the event polarity (i.e., the sign of the brightness change, +1 for brightness increase and -1 for brightness decrease), is the brightness increase amount, is the brightness signal at time , and is the brightness signal at time .
[0049] S32, calculating an actual brightness change amount based on the brightness increase amount.
[0050] Equation (1) is an event generation equation of an ideal sensor. The event polarity of each pixel generated in the time interval is accumulated to generate an image , and the brightness change amount during the time interval is: (2) wherein, is the actual brightness change amount, is a discrete parameter (a pixel on a lattice).
[0051] For smaller , the increment of luminance (2) is due to the moving edge according to the event generation model: (3) The luminance increment is caused by the luminance gradient moving at a speed over a displacement .
[0052] S33, calculate the gradient map of the scene image, based on the current motion parameters and the gradient, calculate the predicted luminance change amount.
[0053] Specifically, the scene image x and y direction gradient maps are extracted respectively, then warped to the specified position according to the parameters of feature tracking, and then the predicted gradient map is obtained according to the assumed motion speed, which is expressed by the following formula: (4) Wherein, is the predicted luminance change amount, is the warped map not used for registration, is the current motion parameter, is the gradient, is the elapsed time.
[0054] S34, difference between the actual luminance change amount and the predicted luminance change amount to obtain the photometric residual, and the photometric residual is taken as the objective function.
[0055] Based on the maximum likelihood method, the difference between the luminance change image generated by the event and the predicted image is used to estimate the motion parameters that best explain the event. The feature tracking problem is proposed using the event and the frame as image registration, and the frame is taken as the characteristic template of the registered event. It is assumed that formula (2) and formula (3) are used when compared, a small patch (P) containing a unique pattern is used, and it is further assumed that the optical flow v is constant for all pixels in the patch (same as the regularization of Lucas and Kanade (1981)). The intensity image generated at t=0 is , and the image generated after t is , so as to find the motion parameters p and the speed v that maximize the similarity between .
[0056] S35, maximize the likelihood of motion parameters p and v ( ) to minimize the norm of the photometric residual: (5) Propose to minimize the difference between the unit norm patches: (6) The above formula cancels C and whose value depends only on the direction of the characteristic velocity v. In this formula, one can consider the same type of parametric algorithm to register the images (projective, affine, etc.). For simplicity, consider a warp caused by a rigid body motion on the image plane: (7) where R(P) is a two-dimensional rotation matrix and t(p) is a translation vector. The objective function (7) can be optimized using the nonlinear least squares framework provided in the Ceres software.
[0057] The optimal motion parameters p and optical flow v are chosen such that the objective function is minimized to obtain the optimized motion parameters and optical flow.
[0058] It does not have the problem of establishing event-feature correspondences, we borrow the typical image registration method of implicit pixel-to-pixel data association to create a convenient image representation from events. Therefore, this method has a small complexity (establishing data association in ICP (Kueng et al. 2016) has quadratic complexity), and is more stable because it is less likely to fall into local minima caused by data association. With the evolution of the optimization iteration, all event correspondences evolve into a single entity according to the evolution of the warped pixel grid.
[0059] According to the method described in the above embodiment, the present embodiment will be further described from the perspective of a visual SLAM front-end optimization device for complex scenes, which can be implemented as an independent entity, or integrated in an electronic device, which can be a terminal, a server, and the like. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, and the like.
[0060] Please refer to Figure 2 , Figure 2 The visual SLAM front-end optimization device for complex scenes provided by the embodiments of the present application is specifically described, which is applied to an electronic device, and can include: An acquisition module, configured to acquire a scene image and acquire event data of a corresponding scene through an event camera; A feature extraction module, configured to perform feature detection based on the event data to obtain an event corner feature flow; An optimization module is configured to extract the time corner point feature flow, generate an actual brightness change amount, generate a predicted brightness change amount based on the scene image, construct an objective function based on the actual brightness change amount and the predicted brightness change amount, minimize the objective function, and obtain optimized motion parameters and an optical flow.
[0061] In a practical implementation, the above modules and / or units can be implemented as independent entities, or can be combined as the same or several entities. The specific implementation of the above modules and / or units can refer to the method embodiments above, and the beneficial effects that can be achieved can also refer to the beneficial effects in the method embodiments above, which will not be repeated here.
[0062] In addition, the embodiments of the present application further provide an electronic device, which can be a computer, a tablet computer, or the like. The electronic device can implement the steps in any of the embodiments of the method for front-end optimization of visual SLAM for complex scenes provided by the embodiments of the present application, and thus can achieve the beneficial effects of any of the methods for front-end optimization of visual SLAM for complex scenes provided by the embodiments of the present application. Details can be found in the above embodiments, which will not be repeated here.
[0063] Figure 3 A specific structure block diagram of an electronic device provided by the embodiments of the present application is shown, which can be used to implement the method for front-end optimization of visual SLAM for complex scenes provided in the above embodiments. The electronic device 500 can be a terminal, a server, or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.
[0064] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.
[0065] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-described embodiments, and the processor 580 can execute various functions and data processing by running the software programs and modules stored in the memory 520, i.e., realize functions such as front camera shooting, processing of the shot image, and switching of display color of the display content on the display screen. The memory 520 can include a high-speed random access memory, and can further include a non-volatile memory such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 520 can further include memories disposed remotely with respect to the processor 580, which can be connected to the electronic device 500 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0066] The input unit 530 can be used to receive inputted digital or character information, and generate a keyboard, a mouse, and the like related to user settings and function control. The display unit 540 can be used to display information inputted by the user or provided to the user, and various graphical user interfaces which can be constituted by graphics, texts, icons, videos, and any combination thereof. The display unit 540 can include a display panel 541, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.
[0067] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can convert received audio data into an electrical signal, transmit the electrical signal to the speaker 561, and convert the electrical signal into a sound signal outputted by the speaker 561; on the other hand, the microphone 562 can convert a sound signal collected into an electrical signal, and the audio circuit 560 can convert the electrical signal into audio data, output the audio data to the processor 580 for processing, and then transmit the audio data to another terminal through the RF circuit 510, or output the audio data to the memory 520 for further processing. The audio circuit 560 can further include an earphone jack to provide communication between an external earphone and the electronic device 500.
[0068] The electronic device 500 can help the user to receive requests, send information, and the like through the transmission module 570 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown, it can be understood that it does not belong to the essential components of the electronic device 500, and can be omitted as needed without changing the essence of the application.
[0069] The processor 580 is a control center of the electronic device 500, which connects various parts of the entire mobile phone through various interfaces and lines, and performs various functions of the electronic device 500 and processes data by running or executing software programs and / or modules stored in the memory 520 and calling data stored in the memory 520, thereby monitoring the entire electronic device. Optionally, the processor 580 can include one or more processing cores; in some embodiments, the processor 580 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. Understandably, the above-mentioned modem processor can also not be integrated into the processor 580.
[0070] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 590 can also include one or more direct or alternating power supplies, recharging systems, power failure detection circuits, power converters or inverters, power state indicators, and any other components.
[0071] Although not shown, the electronic device 500 also includes a camera (such as a front camera or a rear camera), a Bluetooth module, and the like, which are not described here in detail. In this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory and one or more programs stored in the memory and configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Obtaining a scene image and obtaining event data of the corresponding scene through an event camera; Performing feature detection based on the event data to obtain an event corner feature stream; Extracting the time corner feature stream, generating an actual brightness change amount, generating a predicted brightness change amount based on the scene image, constructing a target function based on the actual brightness change amount and the predicted brightness change amount, minimizing the target function, and obtaining optimized motion parameters and optical flow.
[0072] In specific implementation, each of the above modules can be implemented as an independent entity, or can be combined as the same or one or more entities, and the specific implementation of each of the above modules can be referred to the method embodiments described above, which will not be described here.
[0073] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. To this end, the embodiments of the present application provide a storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps of any one of the embodiments of the method for complex scene-oriented visual SLAM front-end optimization provided by the embodiments of the present application.
[0074] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0075] Since the instructions stored in the storage medium can execute the steps in any one of the embodiments of the method for complex scene-oriented visual SLAM front-end optimization provided by the embodiments of the present application, the beneficial effects that can be achieved by any one of the embodiments of the method for complex scene-oriented visual SLAM front-end optimization provided by the embodiments of the present application can be achieved, which are described in detail in the foregoing embodiments and will not be described here.
[0076] The method for complex scene-oriented visual SLAM front-end optimization, the device, the storage medium and the electronic device provided by the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as limiting the present application.
Claims
1. A visual SLAM front-end optimization method for complex scenes, characterized in that, The method includes: Acquire scene images and obtain event data for the corresponding scene through the event camera; Based on the event data, feature detection is performed to obtain the event corner feature flow; Extract the temporal corner feature flow to generate the actual brightness change, generate the predicted brightness change based on the scene image, construct an objective function based on the actual brightness change and the predicted brightness change, minimize the objective function, and obtain the optimized motion parameters and optical flow.
2. The visual SLAM front-end optimization method for complex scenes according to claim 1, characterized in that, Based on the event data, feature detection is performed to obtain the event corner feature flow, including: The events in the event data are filtered, and a G-SAE is constructed. Event-centric local SAEs are extracted using G-SAE to generate local patches, and candidate feature point scores are calculated using these local patches. When the score of the candidate feature point is greater than a preset score threshold, the candidate feature point is determined to be an event feature point; Based on the feature values of the event feature points, a feature grayscale image is constructed, and dilation and non-maximum values are reconciled to obtain the event feature point with the maximum feature value, which is used as the coarse corner point. Create a mask image with a grayscale value of 255, sort the coarse corner points by feature value and traverse them, select the points with a feature value of 255 as fine feature points, and the set of fine feature points is the event corner feature flow.
3. The visual SLAM front-end optimization method for complex scenes according to claim 2, characterized in that, The calculation of candidate feature point scores using local patches is performed using the following formula: in, The scores of candidate feature points For determinant, The trace of the matrix, For the characteristic matrix, These are the eigenvalues of the characteristic matrix M. The matrix elements of the characteristic matrix M, It is a constant; in, These are the gradients of the local patch in the x and y directions, respectively. It is a Gaussian weighted function.
4. The visual SLAM front-end optimization method for complex scenes according to claim 1, characterized in that, Extracting the time corner feature stream to generate the actual brightness change includes: When an event in the time corner feature stream In pixels Triggered in time When the brightness of an event on a pixel increases beyond a set threshold ±C, the increase in brightness is: in, The time when the previous event occurred on the same pixel. ∈{-1, +1} represents the event polarity. The amount of increase in brightness, For time The brightness signal at that time, For time The brightness signal at that time; Calculate the actual change in brightness based on the aforementioned increase in brightness: in, This represents the actual change in brightness. These are discrete parameters.
5. The visual SLAM front-end optimization method for complex scenes according to claim 4, characterized in that, Generating a predicted brightness change based on the scene image includes: Calculate the gradient map of the scene image, and based on the current motion parameters and gradient, calculate the predicted brightness change: in, To predict the amount of change in brightness, Distortion map not used for registration For the current motion parameters, For gradient, This represents the elapsed time.
6. The visual SLAM front-end optimization method for complex scenes according to claim 5, characterized in that, A target function is constructed based on the actual brightness change and the predicted brightness change, including: The difference between the actual brightness change and the predicted brightness change is used to obtain the photometric residual, which is then used as the objective function.
7. The visual SLAM front-end optimization method for complex scenes according to claim 6, characterized in that, Minimize the objective function to obtain the optimized motion parameters and optical flow, including: Minimize the norm of the photometric residual: Propose minimizing the differences between unit specification patches: Consider the distortion caused by rigid body motion on the image plane: Where R(P) is a two-dimensional rotation matrix and t(p) is a translation vector; The optimal motion parameters p and optical flow v are selected to minimize the objective function, thus obtaining the optimized motion parameters and optical flow.
8. A visual SLAM front-end optimization device for complex scenes, characterized in that, include: The acquisition module is used to acquire scene images and obtain event data of the corresponding scene through the event camera; The feature extraction module is used to perform feature detection based on the event data to obtain the event corner feature flow; The optimization module is used to extract the temporal corner feature flow, generate the actual brightness change, generate the predicted brightness change based on the scene image, construct an objective function based on the actual brightness change and the predicted brightness change, minimize the objective function, and obtain the optimized motion parameters and optical flow.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the visual SLAM front-end optimization method for complex scenes as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the visual SLAM front-end optimization method for complex scenes as described in any one of claims 1 to 7.