An event camera based three-dimensional vision calibration and localization method and system

Abstract

The application discloses a three-dimensional vision calibration and positioning method and system based on an event camera, and belongs to the positioning field. An array containing a plurality of unique frequency coded light sources is constructed, a binocular event camera system is built, light signal event streams of the light source array are collected, pixel position, frequency and timestamp information are extracted, pixel coordinate centroids of each light source are determined through frequency and pixel clustering analysis, and a frequency and centroid mapping is established. Auxiliary light sources with the same coding rules are arranged on the surface of a target object, after the binocular system collects light signals of the auxiliary light sources, the centroid coordinates of the light sources are matched according to the mapping relationship. Based on the calibration parameters of the event camera and binocular external parameters, the centroid coordinates are corrected and stereoscopic matched, and the three-dimensional coordinates of the auxiliary light sources are calculated through triangulation. Combined with the known geometric topological relationship of the auxiliary light sources on the surface of the object, the spatial pose of the target object is solved by using a rigid transformation algorithm, and high-precision positioning is completed. The embodiment can improve the calibration precision in a complex application environment.

Description

Technical Field

[0001] This application belongs to the field of positioning, and in particular relates to a three-dimensional visual calibration and positioning method and system based on an event camera. Background Technology

[0002] 3D vision calibration and localization technology is a key foundation for fields such as robot navigation, autonomous driving, and industrial automation. It provides support for precise control and environmental perception by acquiring the spatial pose information of target objects. Traditional methods mostly rely on visible light cameras and calibration boards for parameter calibration and achieve object localization based on technologies such as binocular vision or structured light. However, in dynamic environments or under complex lighting conditions, existing solutions face significant challenges in terms of real-time performance and robustness.

[0003] In existing technologies, 3D calibration typically uses 2D calibration boards such as checkerboards or dot arrays, and solves for camera parameters through multi-angle image acquisition; localization relies on solutions such as binocular stereo matching, structured light projection, or LiDAR. For example, binocular vision calculates depth information through parallax, but requires high computational resources and is sensitive to texture and lighting; structured light is easily affected by ambient light and is difficult to use outdoors; while LiDAR has high accuracy, it is expensive and has difficulty detecting transparent objects.

[0004] Therefore, the above methods have obvious drawbacks: First, traditional calibration boards rely on image feature point recognition, which can easily lead to false detections of feature points when there is uneven lighting or complex backgrounds, affecting calibration accuracy; second, the positioning algorithm based on RGB images has high computational complexity and poor adaptability to highly reflective or transparent objects. Therefore, the existing methods suffer from poor calibration accuracy and inability to adapt to complex application environments. Summary of the Invention

[0005] This application provides a method, system, device, and computer storage medium for three-dimensional visual calibration and positioning based on an event camera, which can improve calibration accuracy in complex application environments.

[0006] Firstly, this application provides a 3D visual calibration and localization method based on an event camera, the method comprising:

[0007] Construct a light source array, which includes multiple independent light sources assigned with unique frequency codes, and build a binocular vision system using at least two event cameras, each equipped with a filter that matches the wavelength of the light signal from the light source array.

[0008] The system acquires the light signal event stream of the light source array through a binocular vision system and extracts the set of valid events from the light signal event stream. The set of valid events includes pixel position, light signal frequency and timestamp.

[0009] By performing cluster analysis on the frequency and pixel position of the optical signal, the centroid of the pixel coordinates of each independent light source is identified, and a mapping relationship between frequency coding and pixel centroid is established.

[0010] Multiple auxiliary light sources are arranged on the surface of the target object, and the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array.

[0011] The target light signal event stream of the target object is acquired by a binocular vision system, the set of effective target events in the target light signal event stream is extracted, and the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the set of effective target events is determined according to the mapping relationship.

[0012] Based on the preset intrinsic parameter matrix and distortion parameters of the event camera, and the extrinsic parameter matrix of the binocular vision system, the set of centroids of pixel coordinates is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation.

[0013] Based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the pose of the target object is calculated through a rigid transformation algorithm to complete the localization. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0014] In one feasible implementation, the set of valid events in the optical signal event stream is extracted, including:

[0015] Create a state matrix with dimensions matching the resolution of the event camera. Each element of the state matrix records the timestamp of the last event and the current polarity state of the corresponding pixel.

[0016] The polarity of a new event in the optical signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. When the polarity state changes, the frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the previous event of the corresponding pixel recorded in the state matrix.

[0017] Valid events whose frequency values ​​are within the preset frequency encoding range of an independent light source are filtered out, and the pixel position, frequency, and timestamp of the valid events are stored in the valid event set.

[0018] In one feasible implementation, by performing cluster analysis on the optical signal frequency and pixel position, the centroid of pixel coordinates for each independent light source is identified, and a mapping relationship between frequency encoding and pixel coordinate centroids is established, including:

[0019] The events in the valid event set are clustered based on the optical signal frequency and pixel location to generate multiple event clusters;

[0020] Remove event clusters from multiple event clusters whose number of elements is below a preset threshold, and retain valid event clusters;

[0021] Calculate the pixel coordinate centroid of the valid event clusters and verify the matching degree between the average frequency of the valid event clusters and the preset frequency encoding of the independent light source;

[0022] Establish a mapping relationship between the centroid coordinates of valid event clusters whose matching degree meets the preset error range and the frequency encoding.

[0023] In one feasible implementation, before correcting and binocular matching the set of pixel coordinate centroids based on the preset intrinsic and distortion parameters of the event camera and the extrinsic matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source using triangulation, the method includes:

[0024] The light source array is moved at multiple angles within the field of view of each event camera to collect the light signal event stream of the light source array in real time and extract the effective recognition results.

[0025] Based on the mapping relationship between frequency coding and pixel coordinate centroid in the effective recognition results, multiple sets of calibration board key point position data are extracted;

[0026] Based on multiple sets of key point location data of calibration boards, the intrinsic parameter matrix and distortion parameters of the event camera are calculated through a planar calibration algorithm;

[0027] When the number of valid recognition results reaches the preset threshold and the key point location data of the calibration board covers the entire field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

[0028] In one feasible implementation, the method further includes:

[0029] A single frequency-coded light source is moved within the overlapping field of view of a binocular vision system, and optical signal event stream data from two event cameras are acquired simultaneously.

[0030] Multiple sets of corresponding point pairs are generated by matching the pixel coordinates of the same light source detected by two event cameras based on frequency coding.

[0031] Based on the corresponding point pairs and intrinsic parameter matrices, the rotation matrix and translation vector are calculated through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system.

[0032] In one feasible implementation, based on the intrinsic parameter matrix, distortion parameters, and extrinsic parameter matrix, the set of pixel coordinate centroids is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated using triangulation, including:

[0033] Using the camera focal length, principal point coordinate parameters and distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the set of centroids of pixel coordinates to generate the corrected pixel coordinates;

[0034] Based on the rotation matrix and translation vector in the extrinsic parameter matrix, stereo epipolar correction is performed on the coordinate system of the event camera in the binocular vision system to achieve coplanar and row alignment of the imaging planes of the event camera, thus forming a stereo correction mapping relationship for binocular vision.

[0035] Based on the stereo correction mapping relationship, the target pixel coordinates are obtained by epipolar constraint matching of the corrected pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras.

[0036] Based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, the depth information is calculated using the triangulation formula, and the three-dimensional spatial coordinates are obtained based on the depth information and the target pixels.

[0037] In one feasible implementation, the pose of the target object is calculated using a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete the localization, including:

[0038] The three-dimensional spatial coordinates of the auxiliary light source are matched with the distribution information of the auxiliary light source to establish the initial pose correspondence.

[0039] The objective function is constructed based on the iterative nearest point algorithm. The rotation matrix and translation vector are optimized by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source.

[0040] When the iterative optimization meets the preset convergence condition, the optimized rotation matrix and translation vector are output to obtain the pose of the target object and complete the localization.

[0041] Secondly, this application provides a 3D visual calibration and localization system based on an event camera, the system comprising:

[0042] A building module is used to build a light source array, which includes multiple independent light sources assigned with unique frequency codes. A binocular vision system is built using at least two event cameras, each equipped with a filter that matches the wavelength of the light signal from the light source array.

[0043] The extraction module is used to acquire the light signal event stream of the light source array through the binocular vision system and extract the set of valid events in the light signal event stream. The set of valid events includes pixel position, light signal frequency and timestamp.

[0044] The building module is also used to identify the centroid of pixel coordinates for each independent light source by performing cluster analysis on the frequency and pixel position of the light signal, and to establish a mapping relationship between frequency coding and pixel coordinate centroid.

[0045] The arrangement module is used to arrange multiple auxiliary light sources on the surface of the target object. The frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array.

[0046] The extraction module is also used to acquire the target light signal event stream of the target object through the binocular vision system, extract the target effective event set in the target light signal event stream, and determine the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the target effective event set according to the mapping relationship.

[0047] The calculation module is used to correct and match the set of centroids of pixel coordinates based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and to calculate the three-dimensional spatial coordinates of the auxiliary light source by triangulation.

[0048] The calculation module is also used to calculate the pose of the target object based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, and to complete the localization by using a rigid transformation algorithm. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0049] Thirdly, this application provides an electronic device, the device including: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the event camera-based three-dimensional visual calibration and positioning method as described in any embodiment of the first aspect.

[0050] Fourthly, this application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the event camera-based 3D visual calibration and localization method as described in any embodiment of the first aspect.

[0051] This application presents a method, system, device, and computer storage medium for 3D visual calibration and localization based on an event camera. This technical solution effectively solves the problems of poor calibration accuracy and inability to adapt to complex application environments inherent in traditional technologies by employing a 3D visual calibration and localization method based on an event camera. Specifically, this solution utilizes the event camera's keen perception of changes in brightness and darkness in the image, combined with an infrared LED strobe calibration board and markers, to achieve accurate calibration under uneven lighting or complex background conditions. This avoids the false detection problems caused by traditional calibration boards relying on image feature point recognition, significantly improving calibration accuracy. Simultaneously, by abandoning complex RGB image processing algorithms, the computational load is greatly reduced, improving real-time performance. Furthermore, it exhibits excellent recognition accuracy and robustness for highly reflective or transparent objects, effectively overcoming the limitations of existing technologies in complex application environments. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a three-dimensional visual calibration and localization method based on an event camera provided in one embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating a three-dimensional localization algorithm based on infrared stroboscopic features from an event camera, provided in one embodiment of this application.

[0055] Figure 3 This is a schematic diagram of the structure of a 3D vision calibration and localization system based on an event camera provided in one embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0057] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0058] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0059] To address the problems of existing technologies, embodiments of this application provide a method, system, device, and computer storage medium for 3D visual calibration and localization based on an event camera. The 3D visual calibration and localization method based on an event camera provided in this application will be described first below.

[0060] Figure 1 A flowchart illustrating a three-dimensional visual calibration and localization method based on an event camera, according to an embodiment of this application, is shown. Figure 1 As shown, the method includes steps S110 to S170.

[0061] S110: Construct a light source array, which includes multiple independent light sources assigned with unique frequency codes, and construct a binocular vision system using at least two event cameras, each equipped with a filter that matches the wavelength of the light signal from the light source array.

[0062] A light source array is a type of calibration device, specifically a stroboscopic calibration board based on infrared LEDs. In 3D vision calibration and positioning systems, constructing the light source array is a crucial step. A light source array consists of multiple independent light sources arranged according to certain rules. These light sources emit light signals encoded with specific frequencies, providing a foundation for subsequent event camera acquisition and feature point recognition.

[0063] The calibration board is fabricated as follows: the light source array typically consists of an m x n array of infrared LEDs (light-emitting diodes). Each infrared LED is assigned a unique ID, and they are encoded sequentially according to these IDs. This encoding method facilitates accurate identification and positioning of each light source during subsequent processing.

[0064] To work with event cameras and filters, infrared LEDs with an emission wavelength of 940nm are typically chosen. Infrared light of this wavelength has low background noise in the environment and is easily filtered by specific filters, thus improving the accuracy of signal detection. The infrared LEDs are fixed to a flat plate to ensure the stability and consistency of the light source array. The selection of the plate should consider factors such as its flatness, heat resistance, and mechanical strength to adapt to different working environments.

[0065] To accurately identify and distinguish each individual light source in a light source array, they need to be frequency-coded. Frequency coding is an effective identification method that allows each light source to be accurately identified in a stream of optical signal events by assigning a unique frequency to each light source.

[0066] A 555 timer configured in astable mode is used to generate a PWM (Pulse Width Modulation) signal to control the flickering of each infrared LED. The PWM signal's duty cycle can be adjusted to control the LED's brightness and flickering frequency. By designing different control circuits, each infrared LED can be made to flicker at a different frequency.

[0067] The frequency of the LED with ID 1 is set to 1kHz. Starting with the LED with ID 2, the frequency of each LED increases by 0.1kHz. For example, the frequency of the LED with ID 2 is 1.1kHz, the frequency of the LED with ID 3 is 1.2kHz, and so on. This frequency encoding method is simple and effective, ensuring that each light source has a unique identifier. The frequency selection should consider the response speed and data processing capabilities of the event camera. Too high a frequency may cause the event camera to fail to detect accurately, while too low a frequency may reduce the real-time performance of the system. The control circuitry is integrated into the PCB (Printed Circuit Board) to control the infrared LED array. The PCB design should consider factors such as circuit stability, reliability, and heat dissipation performance to ensure the long-term stable operation of the light source array.

[0068] Event cameras are one of the core sensors in 3D vision calibration and positioning systems. They can detect changes in brightness and darkness in an image and output data in the form of an event stream. Compared with traditional cameras, event cameras have advantages such as higher dynamic range, lower latency, and lower power consumption.

[0069] An event camera is a biomimetic camera that detects changes in brightness within an image. Its chip differs from traditional CCD and CMOS sensors, employing a completely new architecture. Each pixel operates asynchronously, independently monitoring its own brightness changes. When the brightness change exceeds a certain threshold, it outputs an "event" in the format (x, y, p, t), where x and y represent the pixel position of the event, p is the polarity (1 or 0, indicating whether the point brightens (on event) or darkens (off event), and t is a microsecond-level timestamp. Therefore, its raw data consists of an event stream composed of countless events, rather than image frames. Because it only outputs the changed portions, it significantly reduces the data volume, facilitating transmission and processing, thus laying the foundation for real-time detection.

[0070] An event camera was chosen as the system's sensor. The event camera detects changes in brightness within the image; each pixel works asynchronously, and when the change in brightness exceeds a set threshold, it outputs an event. The event contains information such as pixel location, polarity, and timestamp, providing rich data for subsequent feature point recognition and localization.

[0071] When selecting an event camera, parameters such as resolution, frame rate, dynamic range, and sensitivity should be considered. High-resolution cameras can provide more detailed spatial information, high-frame-rate cameras can capture faster motion changes, while cameras with high dynamic range and sensitivity can maintain stable performance under complex lighting conditions.

[0072] A binocular vision system consists of at least two event cameras. By simulating the binocular vision principle of the human eye, it can acquire depth information of objects, thereby achieving 3D visual calibration and localization. Two event cameras are used to form a binocular stereo vision system. The two cameras capture the same scene from different angles. By comparing the differences between the images captured by the two cameras, the depth information of the objects can be calculated. During installation, the fields of view of both cameras should cover the area to be measured, with sufficient overlap. The existence of this overlap area is fundamental for stereo matching and depth calculation in a binocular vision system.

[0073] Baseline distance and camera angle are important parameters for binocular vision systems. The baseline distance is set to 1 / 4 of the distance to the object being measured for higher accuracy. The camera angle is determined based on the specific object being measured; generally, a larger angle results in higher measurement accuracy but also a closer measurement distance and a smaller measurement range. Therefore, in practical applications, a trade-off must be struck and selected based on the specific scenario.

[0074] An optical filter is an optical element mounted on the lens of an event camera. It filters out light whose wavelength does not match the light signal wavelength of the light source array, thereby reducing background noise interference and improving the detection accuracy of optical signal events. A 940nm bandpass filter is selected because it allows light with a wavelength of 940nm to pass through while blocking other wavelengths. This matches the emission wavelength of the infrared LEDs in the light source array, effectively filtering out background noise. Installing the filter on the camera lens ensures that only light near the 940nm band can pass through. This reduces background and noise interference, ensuring that most of the data acquired by the camera is valid. Increased valid data reduces the total data volume, simplifies data processing, and improves the system's real-time performance and accuracy.

[0075] The specific implementation process is as follows: First, a light source array is constructed. This involves selecting multiple independent light sources and assigning a unique frequency code to each light source. These light sources are arranged according to certain rules to form a light source array. The design of the light source array needs to take into account factors such as the distribution of light sources, the allocation of frequency codes, and the luminous intensity of the light sources to ensure that each light source can be accurately identified and located in subsequent processing.

[0076] Secondly, a binocular vision system is constructed using at least two event cameras. The selection of event cameras needs to consider parameters such as resolution, frame rate, and dynamic range to meet the requirements of 3D visual calibration and localization. The construction of the binocular vision system must ensure that the positions and angles of the two event cameras cover the movement range of the light source array and the target object, in order to acquire sufficient information for 3D reconstruction and localization.

[0077] Finally, install a filter on the event camera that matches the wavelength of the light signal emitted by the light source array. The filter selection needs to match the wavelength of the light signal emitted by the light source array to effectively filter out background noise and improve the detection accuracy of the optical signal event. The filter installation must ensure that it accurately covers the lens of the event camera to avoid light leakage and interference.

[0078] S120: Acquires the light signal event stream of the light source array through a binocular vision system, and extracts the set of valid events in the light signal event stream. The set of valid events includes pixel position, light signal frequency and timestamp.

[0079] Specifically, step S120, which extracts the set of valid events from the optical signal event stream, may include: creating a state matrix, the dimension of which is consistent with the resolution of the event camera, and each element of the state matrix recording the timestamp of the previous event and the current polarity state of the corresponding pixel; comparing the polarity of a new event in the optical signal event stream with the current polarity state of the corresponding pixel in the state matrix, and when the polarity state changes, calculating the frequency value based on the difference between the timestamp of the new event and the timestamp of the previous event of the corresponding pixel recorded in the state matrix; filtering valid events whose frequency values ​​are within the preset frequency encoding range of the independent light source, and storing the pixel position, frequency, and timestamp of the valid events into the set of valid events.

[0080] An optical signal event stream refers to a series of events triggered by light signals emitted by a light source array and captured by an event camera. An event camera is a biomimetic camera capable of detecting changes in brightness within an image. Each pixel works asynchronously, and when the change in brightness exceeds a certain threshold, an event is output. These events contain information such as pixel position, polarity, and timestamp, forming the optical signal event stream. The effective event set is a group of events extracted from the optical signal event stream. These events meet specific conditions, such as their frequency values ​​falling within the preset frequency encoding range of an independent light source. The effective event set contains information such as pixel position, light signal frequency, and timestamp, forming the basis for subsequent processing and analysis. A state matrix is ​​a matrix used to record the state of each pixel, with dimensions consistent with the resolution of the event camera. Each element of the state matrix records the timestamp of the previous event and the current polarity state of the corresponding pixel. Through the state matrix, the state of each pixel can be tracked and updated in real time, thereby accurately extracting the effective event set.

[0081] The specific implementation process is as follows: First, the light signal event stream of the light source array is acquired through a binocular vision system. The binocular vision system consists of at least two event cameras, which are equipped with filters that match the wavelength of the light signal from the light source array to ensure accurate acquisition of the light signal emitted by the light source array. When an individual light source in the light source array flickers, the event camera detects the change in brightness in the image and outputs the corresponding event stream data.

[0082] Next, the set of valid events is extracted from the optical signal event stream. This process includes the following sub-steps: Creating a state matrix: A new state matrix S is created, with dimensions consistent with the resolution of the event camera. Each element of the state matrix is ​​used to record the timestamp of the previous event and the current polarity state of the corresponding pixel. Initially, the polarity state of all pixels can be set to a default value, such as "no change".

[0083] Real-time state matrix updates: Whenever a new event occurs, the polarity of the new event in the optical signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. If the polarity state changes (e.g., from "no change" to "brighter" or "darker"), a frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the previous event recorded in the state matrix for the corresponding pixel. This frequency value reflects the frequency of change of the optical signal at that pixel location.

[0084] Filtering valid events: Based on the calculated frequency values, valid events within the preset frequency encoding range of the individual light sources are filtered out. These valid events correspond to events caused by the flickering of individual light sources in the light source array. The pixel position, frequency, and timestamp information of the valid events are stored in the valid event set.

[0085] Maintaining a valid event set: To ensure the timeliness and accuracy of the valid event set, this set needs to be updated and maintained in real time. For example, a time window can be set to retain only valid events within the most recent time period, while discarding outdated events.

[0086] Through the above sub-steps, a valid event set can be accurately extracted from the light signal event stream acquired by the binocular vision system. This valid event set contains key information about the individual light sources in the light source array, providing an important data foundation for subsequent steps such as feature point recognition, 3D calibration, and localization.

[0087] S130: By performing cluster analysis on the frequency and pixel position of the light signal, the centroid of the pixel coordinates of each independent light source is identified, and a mapping relationship between the frequency code and the centroid of the pixel coordinates is established.

[0088] Specifically, step S130: By performing cluster analysis on the light signal frequency and pixel position, the centroid of the pixel coordinates of each independent light source is identified, and a mapping relationship between the frequency code and the centroid of the pixel coordinates is established. This includes: clustering events in the effective event set based on the light signal frequency and pixel position to generate multiple event clusters; removing event clusters with fewer than a preset threshold number of elements from the multiple event clusters, and retaining the effective event clusters; calculating the centroid of the pixel coordinates of the effective event clusters, and verifying the matching degree between the average frequency of the effective event clusters and the preset frequency code of the independent light source; and establishing a mapping relationship between the centroid coordinates of the effective event clusters with a matching degree within a preset error range and the frequency code.

[0089] Cluster analysis is a statistical method used to group objects in a dataset according to their similarity. In this scheme, cluster analysis is applied to the set of valid events in an optical signal event stream. By analyzing the optical signal frequency and pixel location of the events, similar events are grouped together, thereby identifying the centroid of the pixel coordinates of each independent light source.

[0090] Event clusters are the result of cluster analysis, representing a group of events with similar characteristics. In this scheme, an event cluster refers to a set of events in an optical signal event stream that have similar optical signal frequencies and pixel locations. By analyzing event clusters, the centroid of pixel coordinates for each independent light source can be identified.

[0091] The centroid of pixel coordinates is the average of the pixel positions of all events in an event cluster, representing the center position of that event cluster in the image. In this scheme, the centroid of pixel coordinates is used to establish the mapping relationship between frequency coding and pixel coordinates, providing a foundation for subsequent three-dimensional spatial coordinate calculations.

[0092] The mapping relationship refers to the one-to-one correspondence between frequency codes and pixel coordinate centroids. In this scheme, after identifying the pixel coordinate centroid of each independent light source through cluster analysis, it is necessary to establish a mapping relationship between the centroid and the light source frequency code so that the corresponding pixel coordinate centroid can be quickly located based on the frequency code in subsequent processing.

[0093] The specific implementation process is as follows: First, cluster analysis is performed on the events in the valid event set. This step aims to group events with similar light signal frequencies and pixel locations into one category, thereby identifying all events generated by each independent light source in the image. Cluster analysis can employ various algorithms, such as K-means and DBSCAN, and the specific algorithm should be selected based on the actual data and scenario.

[0094] During cluster analysis, multiple event clusters are generated. Each event cluster represents all events generated by a single light source in the image. However, not all event clusters are valid. Some event clusters may contain only a small number of events, which may be due to noise or interference. Therefore, it is necessary to filter the event clusters, removing those with fewer than a preset threshold number of elements, and retaining only the valid event clusters.

[0095] Next, the centroid of the pixel coordinates for the valid event clusters is calculated. The centroid is the average of the pixel positions of all events in the event cluster, representing the center position of that event cluster in the image. By calculating the centroid, the position of each individual light source in the image can be accurately located.

[0096] Next, the matching degree between the average frequency of the valid event cluster and the preset frequency code of the individual light source is verified. Since each individual light source is assigned a unique frequency code, the average frequency of the valid event cluster can be calculated and compared with its preset frequency code to verify whether the event cluster is indeed generated by a certain individual light source. If the matching degree is within the preset error range, the event cluster is considered valid, and a mapping relationship is established between its centroid coordinates and the frequency code.

[0097] Finally, a mapping relationship is established between the centroid coordinates of valid event clusters whose matching degree meets the preset error range and their frequency codes. This step is the core objective of step S130. By establishing the mapping relationship between the frequency codes and the pixel coordinate centroids, a foundation can be provided for subsequent three-dimensional spatial coordinate calculations. In subsequent processing, when it is necessary to locate a specific light source, the corresponding pixel coordinate centroid can be found based on its frequency code, and the position of the light source in the image can be quickly located.

[0098] S140: Multiple auxiliary light sources are arranged on the surface of the target object. The frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array.

[0099] Auxiliary light sources refer to multiple light sources arranged on the surface of a target object. These light sources have the same frequency coding rules as the individual light sources in the light source array. The function of auxiliary light sources is to form feature points on the surface of the target object so that they can be acquired and located by a binocular vision system.

[0100] Frequency coding rules refer to the rules for assigning a unique frequency code to each individual light source in the light source array. In step S140, the frequency codes of the auxiliary light sources need to be the same as the frequency coding rules of the light source array to ensure that each auxiliary light source can be accurately identified and located in subsequent processing.

[0101] The specific implementation process is as follows: First, auxiliary light sources need to be placed at suitable locations on the surface of the target object. These locations should be able to fully reflect the shape and orientation of the target object, while avoiding mutual interference and occlusion. The number and arrangement of auxiliary light sources should be determined based on the complexity of the target object and the positioning accuracy requirements.

[0102] When arranging auxiliary light sources, it is necessary to ensure that the frequency encoding of each auxiliary light source follows the same rules as the individual light sources in the light source array. This means that each auxiliary light source should also be assigned a unique frequency code, and this code should correspond to the code of the individual light source at the corresponding position in the light source array. By maintaining consistency in the frequency encoding rules, it can be ensured that the binocular vision system can accurately identify and locate each auxiliary light source during subsequent processing.

[0103] To achieve this goal, an auxiliary light source can be created using the same light source and control circuitry as the light source array. Specifically, the same infrared LEDs as those in the light source array can be used, and they can be assigned the same frequency codes as the independent light sources at corresponding positions in the light source array. Simultaneously, it is necessary to ensure that the control circuitry of the auxiliary light source operates synchronously with the control circuitry of the light source array to guarantee the consistency and accuracy of the frequency codes.

[0104] S150: Acquire the target light signal event stream of the target object through a binocular vision system, extract the target valid event set in the target light signal event stream, and determine the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the target valid event set according to the mapping relationship.

[0105] The target optical signal event stream refers to a series of events triggered by the light signals emitted by auxiliary light sources arranged on the surface of a target object, which are acquired by a binocular vision system. These events contain information such as pixel position, light signal frequency, and timestamps, forming the target optical signal event stream.

[0106] The target valid event set is a group of events extracted from the target optical signal event stream. These events meet specific conditions, such as their frequency values ​​falling within the preset frequency encoding range of the auxiliary light source. The target valid event set contains information such as pixel position, optical signal frequency, and timestamp, and forms the basis for subsequent processing and analysis.

[0107] The set of pixel centroids refers to the set of pixel centroids in the image identified by cluster analysis of events in the target's valid event set. These pixel centroids represent the position of each auxiliary light source in the image and are crucial information for subsequent 3D spatial coordinate calculations and target object localization.

[0108] The specific implementation process is as follows: First, a binocular vision system is used to acquire the target object's light signal event stream. The binocular vision system consists of at least two event cameras, which are equipped with filters that match the wavelength of the light signal from the light source array to ensure accurate acquisition of the light signal emitted by the auxiliary light source arranged on the surface of the target object. When the auxiliary light source flickers, the event cameras detect changes in brightness in the image and output the corresponding event stream data.

[0109] Next, a set of valid target events is extracted from the target optical signal event stream. This process is similar to the process of extracting the set of valid events in step S120, including steps such as creating a state matrix, updating the state matrix in real time, and filtering valid events. Through these steps, a set of valid target events that meet specific conditions can be accurately extracted from the target optical signal event stream.

[0110] Finally, the set of pixel centroids corresponding to the frequency codes of each auxiliary light source in the target valid event set is determined based on the mapping relationship. This step utilizes the mapping relationship between frequency codes and pixel centroids established in step S130. By looking up a table or calculating, the corresponding pixel centroid can be quickly located based on the frequency code of each event in the target valid event set. These pixel centroids form a set of pixel centroids, providing crucial information for subsequent 3D spatial coordinate calculations and target object localization.

[0111] S160: Based on the preset intrinsic parameter matrix and distortion parameters of the event camera, as well as the extrinsic parameter matrix of the binocular vision system, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation.

[0112] Specifically, step S160: Based on the intrinsic parameter matrix, distortion parameters, and extrinsic parameter matrix, the set of pixel coordinate centroids is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated using triangulation, including:

[0113] Using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the centroid set of pixel coordinates to generate corrected pixel coordinates. Based on the rotation matrix and translation vector in the extrinsic parameter matrix, stereo epipolar correction is performed on the coordinate system of the event camera in the binocular vision system to achieve coplanar and row alignment of the imaging planes of the event cameras, forming a binocular stereo correction mapping relationship. According to the binocular stereo correction mapping relationship, epipolar constraint matching is performed on the corrected pixel coordinates to obtain the target pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras. Based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, depth information is calculated using triangulation formulas, and three-dimensional spatial coordinates are obtained based on the depth information and the target pixel.

[0114] Nonlinear distortion correction refers to the process of correcting the set of centroids of pixel coordinates using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix. Due to factors such as lens manufacturing errors and installation errors, images captured by an event camera will exhibit distortion. The purpose of nonlinear distortion correction is to eliminate this distortion, so that the corrected pixel coordinates can more accurately reflect the actual position of the target object.

[0115] Stereo epipolar calibration refers to the process of correcting the event camera coordinate system of a binocular vision system based on the rotation matrix and translation vector in the extrinsic parameter matrix. The purpose of stereo epipolar calibration is to make the imaging planes of the two cameras coplanar and aligned, thereby simplifying the stereo matching process and improving matching accuracy and efficiency.

[0116] Epipolar constraint matching refers to the process of matching corrected pixel coordinates based on the stereo calibration mapping relationship. Epipolar constraint matching ensures that matching point pairs from the same auxiliary light source in different event camera images lie on the same epipolar line, thereby improving the accuracy and robustness of the matching.

[0117] Triangulation refers to a method that calculates the depth information of an auxiliary light source based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, and then converts this information into three-dimensional spatial coordinates. Triangulation is a commonly used depth calculation method in binocular vision systems. It utilizes the geometric relationship between disparity and depth to accurately calculate the three-dimensional spatial coordinates of a target object.

[0118] The specific implementation process is as follows: First, using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the set of centroids of pixel coordinates. Due to lens distortion, directly using uncorrected pixel coordinates for 3D spatial coordinate calculation will introduce errors. Therefore, nonlinear distortion correction is needed to eliminate these errors and generate corrected pixel coordinates.

[0119] Next, based on the rotation matrix and translation vector in the extrinsic parameter matrix, stereo epipolar correction is performed on the event camera coordinate system of the binocular vision system. The purpose of stereo epipolar correction is to make the imaging planes of the two cameras coplanar and aligned, thereby simplifying the stereo matching process. Through stereo epipolar correction, the stereo correction mapping relationship of the binocular system can be obtained, providing a foundation for subsequent epipolar constraint matching.

[0120] Then, based on the stereo calibration mapping relationship, epipolar constraint matching is performed on the calibrated pixel coordinates. Epipolar constraint matching ensures that matching point pairs of the same auxiliary light source in different event camera images lie on the same epipolar line, thereby improving the accuracy and robustness of the matching. Through epipolar constraint matching, matching point pairs of the same auxiliary light source in different event camera images can be obtained, and the horizontal disparity value can be calculated.

[0121] Finally, based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, depth information is calculated using triangulation formulas. Then, based on the depth information and the target pixel coordinates, the three-dimensional spatial coordinates of the auxiliary light source are obtained. Triangulation utilizes the geometric relationship between disparity and depth to accurately calculate the three-dimensional spatial coordinates of the target object.

[0122] In one embodiment, to further improve the accuracy and robustness of 3D spatial coordinate calculation, multi-view geometric constraints and global optimization methods can be employed. Specifically, multiple event cameras can be used to acquire data about the target object from different perspectives, obtaining sets of pixel coordinate centroids from multiple viewpoints. Then, these sets of pixel coordinate centroids are jointly matched and optimized using multi-view geometric constraints to eliminate mismatches and noise interference. Finally, a global optimization method is used to jointly optimize all matching point pairs to obtain more accurate and robust 3D spatial coordinates. This method can further improve the accuracy and robustness of event camera-based 3D visual calibration and localization methods.

[0123] S170: Based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the pose of the target object is calculated through a rigid transformation algorithm to complete the localization. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0124] Specifically, step S170: Based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the pose of the target object is calculated using a rigid transformation algorithm to complete the localization. This includes: matching the feature points of the three-dimensional spatial coordinates of the auxiliary light source with the distribution information of the auxiliary light source to establish an initial pose correspondence; constructing an objective function based on the iterative nearest point algorithm, optimizing the rotation matrix and translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source; and outputting the optimized rotation matrix and translation vector when the iterative optimization meets the preset convergence condition to obtain the pose of the target object and complete the localization.

[0125] A rigid transformation algorithm refers to a transformation in which an object undergoes only rotation and translation in space, without deformation. The rigid transformation algorithm utilizes this transformation characteristic to determine the object's pose in space by calculating the rotation matrix and translation vector. In this scheme, the rigid transformation algorithm is used to calculate the target object's pose based on the three-dimensional spatial coordinates and distribution information of the auxiliary light source.

[0126] Feature point matching refers to matching feature points, which are unique and repeatable points in an image, such as corner points and edge points. Feature point matching establishes spatial relationships between feature points in different images by mapping them to each other. In this scheme, feature point matching is used to establish the correspondence between the auxiliary light source and the surface of the target object in three-dimensional space.

[0127] The Iterative Closest Point (ICP) algorithm is a method for 3D point cloud registration. It iteratively searches for the closest point pair between two point clouds and calculates the rotation matrix and translation vector to minimize the spatial distance residual between the two point clouds. In this scheme, the ICP algorithm is used to optimize the rotation matrix and translation vector to improve the accuracy of the target object's pose calculation.

[0128] The spatial distance residual refers to the difference between the actual spatial distance between two points and the calculated spatial distance. In this scheme, the spatial distance residual is the objective function for optimizing the rotation matrix and translation vector. By minimizing the spatial distance residual, a more accurate rotation matrix and translation vector can be obtained.

[0129] The specific implementation process is as follows: First, feature point matching is performed between the three-dimensional spatial coordinates of the auxiliary light source and its distribution information. The distribution information of the auxiliary light source includes its geometric topological relationship and relative position information on the target object's surface. This information can be obtained through pre-measurement or design. The purpose of feature point matching is to establish the correspondence between the auxiliary light source in three-dimensional space and the target object's surface, providing a foundation for subsequent pose calculations. Through feature point matching, the initial pose correspondence of the auxiliary light source on the target object's surface can be obtained.

[0130] Next, the objective function is constructed based on the Iterative Closest Point Algorithm (ILAB). The ILAB is an algorithm for optimizing rotation matrices and translation vectors. It optimizes the rotation matrix and translation vectors by minimizing the spatial distance residuals between the 3D spatial coordinates of the auxiliary light source and the corresponding coordinates in the auxiliary light source distribution information. In this scheme, the objective function is constructed as the sum of the spatial distance residuals between the 3D spatial coordinates of the auxiliary light source and the corresponding coordinates in the auxiliary light source distribution information, as shown in the following formula:

[0131]

[0132] Where E is the spatial distance residual, R is the rotation matrix, T is the translation vector, Pi is the three-dimensional spatial coordinates of the auxiliary light source, Qi is the corresponding coordinate in the auxiliary light source distribution information, and n is the number of auxiliary light sources.

[0133] Then, the rotation matrix and translation vector are optimized by minimizing the spatial distance residual. This step is the core of the iterative closest point algorithm, which iteratively updates the rotation matrix and translation vector to gradually reduce the spatial distance residual. In each iteration, the algorithm calculates the spatial distance residual under the current rotation matrix and translation vector, and updates the rotation matrix and translation vector according to the magnitude and direction of the residual. This process continues until a preset convergence condition is met.

[0134] Finally, when the iterative optimization meets the preset convergence conditions, the optimized rotation matrix and translation vector are output to obtain the pose of the target object, thus completing the localization. The preset convergence conditions can be, for example, the number of iterations reaching its maximum value, the spatial distance residual being less than a certain threshold, or the change in the rotation matrix and translation vector being less than a certain threshold. When any of these conditions is met, the iterative optimization can be considered to have converged, and the rotation matrix and translation vector at this point represent the pose of the target object. Through this process, precise localization of the target object can be achieved.

[0135] This embodiment effectively solves the problems of poor calibration accuracy and inability to adapt to complex application environments in traditional technologies by adopting a 3D visual calibration and positioning method based on event cameras. Specifically, this solution utilizes the event camera's keen perception of changes in brightness and darkness in the image, combined with an infrared LED strobe calibration board and markers, to achieve accurate calibration under uneven lighting or complex background conditions. This avoids the false detection problem caused by traditional calibration boards relying on image feature point recognition, significantly improving calibration accuracy. At the same time, by abandoning complex RGB image processing algorithms, the computational load is greatly reduced, real-time performance is improved, and excellent recognition accuracy and robustness are demonstrated for highly reflective or transparent objects, effectively overcoming the limitations of existing technologies in complex application environments.

[0136] In one feasible implementation, before step S160: correcting and binocular matching of the pixel coordinate centroid set based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source by triangulation, the method includes:

[0137] The light source array is moved at multiple angles within the field of view of each event camera to collect the light signal event stream of the light source array in real time and extract the valid recognition results. Based on the mapping relationship between the frequency code and the centroid of the pixel coordinates in the valid recognition results, multiple sets of calibration board key point position data are extracted. Based on the multiple sets of calibration board key point position data, the intrinsic parameter matrix and distortion parameters of the event camera are calculated through a planar calibration algorithm. When the number of valid recognition results reaches a preset threshold and the calibration board key point position data covers the entire field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

[0138] An event camera is a biomimetic camera capable of detecting changes in brightness within an image. Each pixel can independently monitor its own brightness changes, and when the brightness change exceeds a certain threshold, it outputs an event containing position, polarity, and a timestamp. This characteristic gives event cameras significant advantages in high-speed motion and low-light environments. The light source array, in this embodiment, refers to a strobe calibration board based on infrared LEDs. The infrared LEDs are arranged in a specific array and can strobe; each LED has a unique strobe frequency for subsequent identification and positioning. The optical signal event stream refers to the data stream composed of events generated by brightness changes captured by the event camera. These events include the LED's strobe position (x, y coordinates), polarity (brightening or darkening), and timestamp, serving as the foundational data for subsequent processing. Frequency encoding refers to the unique strobe frequency of each infrared LED, which is used as an encoding to identify the LED's position in the array. By decoding this frequency information, the LED's identity and position can be determined.

[0139] The centroid of a pixel coordinate refers to the pixel coordinates that represent the center of an event cluster, obtained through a clustering algorithm. In the event stream captured by an event camera, events generated by the same LED flashing will form an event cluster. By calculating the centroid of this event cluster, the precise position of the LED in the image can be obtained.

[0140] The intrinsic parameter matrix and distortion parameters refer to the camera's internal parameters and the parameters caused by lens distortion. The intrinsic parameter matrix includes the camera's internal parameters such as focal length and principal point, while the distortion parameters describe the lens distortion. These parameters are fundamental to subsequent tasks such as image correction and 3D localization.

[0141] The specific process is as follows: First, the infrared LED-based stroboscopic calibration board (light source array) is moved at multiple angles within the field of view of each event camera. The purpose of this step is to ensure that all LED points on the calibration board can be captured from different camera perspectives, thereby obtaining comprehensive calibration data. By moving the board at multiple angles, the entire field of view of the camera can be covered, taking into account distortion and perspective changes at different perspectives.

[0142] Next, the event camera captures the brightness changes caused by the flashing of infrared LEDs on the calibration board in real time, forming a stream of optical signal events. These events contain the position, polarity, and timestamp information of the LED flashes, which are the basic data for subsequent processing. The characteristics of the event camera enable it to capture clear event streams in high-speed motion and low-light environments.

[0143] Then, based on the frequency information and pixel coordinates in the event stream, valid recognition results are extracted using a specific algorithm (such as clustering). Valid recognition results refer to the set of events that successfully identify and match LED points with a predetermined frequency code. This step allows for the filtering of events belonging to LED points on the calibration board, providing accurate data for subsequent processing.

[0144] Subsequently, based on the mapping relationship between frequency codes and pixel centroids in the valid recognition results, multiple sets of calibration board keypoint location data are extracted. Each keypoint location data includes the pixel coordinates of that point in the image and its corresponding frequency code. This keypoint location data forms the basis for subsequent calculations of the camera intrinsic parameter matrix and distortion parameters.

[0145] Then, based on multiple sets of key point location data from the calibration board, the intrinsic parameter matrix and distortion parameters of the event camera are calculated using a planar calibration algorithm (such as Zhang's calibration method). The intrinsic parameter matrix contains internal parameters of the camera, such as focal length and principal point, while the distortion parameters describe lens distortion. Through the planar calibration algorithm, precise intrinsic parameters and distortion parameters of the camera can be obtained, providing an accurate foundation for subsequent image correction and 3D localization tasks.

[0146] Finally, when the number of valid recognition results reaches a preset threshold (e.g., more than 20 times), and the calibration board key point location data covers the entire field of view of the event camera, the calculated intrinsic parameter matrix and distortion parameters are output. These parameters will be used for subsequent image correction and 3D localization tasks to ensure the accuracy and stability of the system.

[0147] In one feasible implementation, the method further includes: moving a single frequency-coded light source within the overlapping field of view of the binocular vision system and simultaneously acquiring optical signal event stream data from two event cameras; matching the pixel coordinates of the same light source detected by the two event cameras according to the frequency coding to generate multiple sets of corresponding point pairs; and calculating the rotation matrix and translation vector through essential matrix decomposition based on the corresponding point pairs and the intrinsic parameter matrix to obtain the extrinsic parameter matrix of the binocular vision system.

[0148] In this embodiment, a frequency-coded light source refers to an infrared LED light source capable of emitting light at a specific frequency. Each light source has a unique flicker frequency, and this frequency information is used as an encoding to identify the light source. By detecting this frequency information, different light sources can be distinguished. A binocular vision system is a system consisting of two cameras that capture images of the same object and calculate the parallax, using triangulation to calculate the object's 3D information. Binocular vision systems can acquire depth information of objects and are the foundation for achieving 3D visual positioning.

[0149] In a binocular vision system, the essential matrix is ​​a 3x3 matrix used to describe the relative pose (rotation and translation) between the two cameras and the geometry of the scene. By decomposing the essential matrix, we can obtain the rotation matrix and translation vectors, thus determining the relative positional relationship between the two cameras. The rotation matrix describes the rotational relationship between the two cameras in 3D space. It allows us to transform a point in one camera coordinate system to another. The translation vector describes the translational relationship between the two cameras in 3D space. The translation vector represents the displacement between the origins of the two camera coordinate systems. The extrinsic parameter matrix, also in a binocular vision system, is composed of the rotation matrix and translation vectors and describes the relative pose between the two cameras. The extrinsic parameter matrix is ​​one of the key parameters for achieving 3D visual localization.

[0150] Specifically, the steps include: First, install two event cameras at predetermined positions and angles, ensuring that their fields of view overlap. This is the foundation for binocular visual positioning; only when the fields of view of the two cameras overlap can different perspective images of the same object be captured, thereby calculating parallax and depth information.

[0151] Next, both cameras are activated to synchronously acquire optical signal event stream data. Since the event camera only outputs the changing parts, the data volume is small and easy to process in real time. Synchronously acquiring event stream data from both cameras ensures their temporal synchronization, providing an accurate foundation for subsequent data processing.

[0152] Then, a frequency-coded infrared LED light source is moved within the overlapping field of view. This light source emits light at a specific frequency, and both cameras detect the event stream data of the light source in real time, identifying the light source based on the frequency coding. Frequency coding ensures that both cameras detect the same light source, providing a basis for subsequent data matching.

[0153] Next, by comparing the event stream data detected by the two cameras, the pixel coordinates of the same light source in both cameras are matched, generating multiple sets of corresponding point pairs. These corresponding point pairs are the basic data for subsequent calculation of the essential matrix, representing the positional relationship of the same light source under different viewpoints in the two cameras.

[0154] Finally, based on the corresponding point pairs and intrinsic parameter matrices, the rotation matrix and translation vector are calculated through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system. Given the intrinsic parameter matrices of the two cameras (obtained through the previous calibration process), the essential matrix can be calculated using the corresponding point pairs and intrinsic parameter matrices. Singular value decomposition (SVD) is performed on the essential matrix to obtain candidate solutions for the rotation matrix and translation vector. Triangulation is used to determine whether the depth of the point reprojection onto the camera is positive; solutions with positive depth are retained to obtain the final rotation matrix R and translation vector t. The rotation matrix R and translation vector t together constitute the extrinsic parameter matrix of the binocular vision system, describing the relative pose between the two cameras.

[0155] In one implementation, this embodiment provides a complete 3D visual calibration and localization method based on an event camera, the details of which are as follows:

[0156] This method achieves 3D vision calibration and localization based on an event camera, specifically involving a calibration method and device, as well as a localization method based on key point recognition.

[0157] First, the calibration board is fabricated. The aforementioned calibration device refers to a stroboscopic calibration board based on infrared LEDs. Its specific fabrication method is as follows:

[0158] Step 1: Arrange (m*n) infrared LEDs in an array of m rows and n columns and fix them on a flat plate. Encode each LED sequentially starting from 1, assigning each LED a unique ID. The infrared LED emission wavelength is selected as 940nm.

[0159] Step 2: Design a control circuit to implement the strobe function of the infrared LEDs and control the frequency. Specifically, each infrared LED is controlled using a 555 timer. The 555 timer is configured in astable mode to generate a PWM signal. In this mode, the output frequency is determined by two resistors (R1 and R2) and one capacitor (C1). The frequency formula is: f = 1.44 / ((R1 + 2R2) * C1).

[0160] Step 3: Integrate the control circuit described in Step 1 onto the PCB board to achieve control of the infrared LED array.

[0161] Step 4: Set the frequency of the lights in the array. Set the frequency of the light with ID 1 to 1 kHz. Starting from the light with ID 2, increase the frequency of each light by 0.1 kHz. The last light with ID (m*n) has a frequency of 1 + (m*n-1)*0.1 kHz.

[0162] By following the steps above, an infrared LED array with ID encoding can be obtained, where each ID LED has a unique frequency. This device can be used as a visual calibration board for calibrating event cameras.

[0163] Next, 3D visual calibration is performed. The specific steps for calibration are as follows:

[0164] Step 1: Install the filter. Install a 940nm bandpass filter onto the camera lens to ensure that only light near the 940nm wavelength can pass through. This greatly reduces background and noise interference, ensuring that the majority of the data acquired by the camera is valid data, reducing the total data volume and laying the foundation for real-time computation.

[0165] Step 2: Install the cameras according to the actual test scenario. This method requires two event cameras to form a binocular stereo vision system. During installation, ensure that the fields of view of both cameras cover the test area with sufficient overlap. The baseline distance and camera angle are determined by the distance and range of the object being measured. The baseline distance is set to 1 / 4 of the distance to the object, which is larger than the baseline distance of a conventional binocular system to obtain higher accuracy. Regarding the camera angle, generally speaking, the larger the camera angle, the closer the measurement distance, the smaller the measurement range, and the higher the measurement accuracy; conversely, the smaller the angle, the farther the measurement distance, the larger the measurement range, and the lower the measurement accuracy. The angle should be determined based on the actual conditions of the object being measured. After installation, the cameras must not be moved or the angle adjusted during the calibration and measurement processes.

[0166] Step 3: Perform intrinsic parameter calibration on both event cameras. The method for calibrating the intrinsic parameters of a single camera is to move the frequency-encoded infrared LED calibration board within the camera's field of view, calculate the position of the calibration board in the image in real time based on the event stream data captured by the camera, and finally calculate the camera's intrinsic parameter matrix and distortion parameters according to Zhang's calibration method.

[0167] The specific steps for calculating and calibrating the calibration board position using event flow are as follows:

[0168] (1) Create a new matrix representing the state of each pixel, called the state matrix S. The shape and size of the state matrix are equal to the camera resolution, where each element corresponds to a pixel at the corresponding position of the camera. The state matrix is ​​a 3-channel matrix, and each pixel stores 3 pieces of information: the timestamp of the last on event; the timestamp of the last off event; and the current state, where 1 or 0 indicates on or off.

[0169] (2) Create a valid event set E. Whenever a new event occurs, based on the state matrix, compare the polarity of the new event with the current state of the pixel position in the state matrix. If the state changes, calculate the frequency f_xy = 1 / (t_2 - t_1). xy The frequency represents the location of the event, t2 represents the timestamp of the new event, and t1 represents the timestamp of the previous event of this polarity. If the frequency satisfies 0.9 (kHz)... <f xy If the value is less than 1 + 0.1m*n (kHz), then this event is added to the valid event set V. Each event in V needs to store the following attributes: pixel position x, y, frequency f, and timestamp t. Subsequently, regardless of whether the state changes, the elements at the corresponding pixel positions in the state matrix are updated. The current state in the matrix is ​​replaced with the polarity of the new event, and the timestamp of the new event replaces the timestamp of the previous event with this polarity in the matrix.

[0170] (3) Set E stores the most recent valid events. Events with older timestamps exceeding a threshold are discarded to ensure real-time performance. The time threshold can be set according to requirements, typically between 1ms and 15ms. Therefore, E is updated in real time based on the event stream.

[0171] (4) Cluster all events in V at the latest time. Methods such as k-means, DBSCAN, and hierarchical clustering can be selected, clustering based on the x, y, and f of the events. After clustering, discard clusters with fewer than 5 elements to obtain several valid clusters. If the number of valid clusters is (m*n), and the average frequency of events within each cluster corresponds one-to-one with the preset frequency, with a frequency error not exceeding 0.02kHz, then the calibration board is considered successfully identified. Calculate the centroid of the pixel coordinates of each cluster sequentially according to frequency order as the keypoint coordinates, obtaining the pixel coordinates of each infrared LED, and record them by number.

[0172] (5) After the calibration board is successfully identified for the first time, the coordinates of the center point of the calibration board are recorded and stored in set C. Each subsequent successful identification requires comparison with all the center point coordinates already recorded in C: the Euclidean distance between the newly identified center point coordinates and each coordinate in C is calculated. If any distance is less than a threshold, the identification result is discarded. If the distance is greater than or equal to the threshold, the identification is considered valid, the key point position is recorded, and the center point coordinates are stored in set C. This operation avoids multiple identifications of similar positions.

[0173] (6) Move the calibration board within the camera's field of view, identify its position in real time, and continuously accumulate valid recognitions. When the coordinates of the center points effectively recognized in set C evenly cover the entire image, and the number of valid recognitions is greater than 20, end the shooting.

[0174] (7) Using the key point location data of the calibration board in all the above valid identifications, perform intrinsic parameter calibration according to Zhang's calibration method, and calculate the camera's intrinsic parameter matrix and distortion parameters.

[0175] Step 3: Extrinsic parameter calibration. Obtain the rotation matrix R and the normalized translation vector t through the correspondence between point pairs, and then determine the true translation vector t through scale calibration. true The specific implementation method is as follows:

[0176] (1) Use a single strobe infrared LED as a feature point and move to capture images in the overlapping field of view of two cameras.

[0177] (2) Two cameras take real-time pictures, and the feature points are detected and located according to the recognition method described in step 2. The coordinates of the feature points captured by the two cameras at the same time are saved in pairs.

[0178] (3) Move the feature points to collect more point pairs. The feature points should appear as evenly as possible in all positions of the field of view, and the number of point pairs should be greater than 200.

[0179] (4) Solve for the essential matrix E based on the corresponding point pairs and the intrinsic parameter matrices of the two cameras.

[0180] (5) Perform singular value decomposition (SVD) on the essential matrix E. The decomposition formula is E = U∑V T Where U and V are orthogonal matrices, and ∑ is the singular value matrix. The rotation vector may be UWV T or UW T V T The translation vector is the third column vector of U, and its sign can be positive or negative. By triangulation, we determine whether the depth of the point reprojected onto the camera is positive. We retain a set of solutions with positive depth to obtain the rotation matrix R and the translation vector t.

[0181] (6) The t obtained at this point is a normalized translation vector, which only has direction information and no scale information. To determine the scale information, scale calibration is required. The specific method is as follows: use a stroboscopic infrared LED calibration board, where the distance d between each pair of key points on the calibration board is known. ij The three-dimensional coordinates of all points on the calibration board are obtained by triangulation, and the distances between each pair of points are calculated. The loss function represents the error between the recovered distance and the true distance. Where s is the scaling factor. Using least squares or other optimization methods, we find the s that minimizes the loss function, and then the scaled translation vector t is obtained. true =s·t. The calibration board used in this step can be selected according to the actual scenario, and the size of the calibration board should be as large as possible without exceeding the field of view.

[0182] Next, 3D object localization is performed. A strobe infrared LED is used as a marker to assist in localization. The specific method is as follows:

[0183] Step 1: Based on the size and shape of the object being measured, select several strobe infrared LEDs as markers and place them on the surface of the object. Encode them sequentially starting from 1, assigning each LED a unique ID. Set the frequency of the LED with ID 1 to 1 kHz. Starting with the LED with ID 2, increase the frequency of each LED by 0.1 kHz. The last LED has ID number N and a frequency of 1 + (N-1) * 0.1 kHz.

[0184] Step 2: Position the infrared LEDs according to the method described in Step 2 of the calibration section to obtain the pixel coordinates of each LED in the two cameras.

[0185] Step 3: Based on the camera's intrinsic and extrinsic parameters and distortion parameters, perform distortion correction and stereo correction on the pixel coordinates.

[0186] Step 4: Based on the detected frequency, match the key points detected in the left and right cameras to obtain N pairs of points.

[0187] Step 5: Obtain the depth Z using triangulation. The calculation formula is Z = (f·B) / d, where f is the focal length, B represents the baseline distance, and d represents the pixel difference between corresponding points on the two cameras. Finally, convert the pixel coordinates to three-dimensional coordinates.

[0188] Step 6: Based on the three-dimensional coordinates of all points, the pose and surface morphology of the object under test can be obtained, thus realizing 3D positioning.

[0189] Figure 2 A flowchart illustrating a three-dimensional localization algorithm based on infrared stroboscopic features from an event camera, according to an embodiment of this application, is shown. Figure 2 As shown, the method includes steps S210 to S280.

[0190] In one embodiment, the implementation of the 3D localization algorithm based on the infrared stroboscopic features of an event camera is as follows:

[0191] S210: New event input: Receives a real-time light change event stream captured by the event camera as the algorithm input source.

[0192] S220: State change judgment: By comparing the current event with the recorded value of the state matrix, if the difference exceeds the preset threshold (e.g., ΔI>10%), it is judged as a state change event.

[0193] S230: State Matrix Update: For events that have not reached the threshold, directly update the corresponding pixel state value to maintain spatial continuity recording.

[0194] S240: Strobe Period Calculation: For sudden state events, the flicker frequency of the infrared LED is calculated using the timestamp difference (f = 1 / Δt).

[0195] S250: Valid event filtering: Filters out interference events based on a preset flicker frequency range (e.g., 8-12Hz) and retains valid events that match the characteristics of the calibration board LED.

[0196] S260: Spatiotemporal clustering analysis: The density-based DBSCAN algorithm is used to perform spatial proximity and temporal continuity clustering on valid events.

[0197] S270: Frequency Feature Matching: Compare the cluster center frequency with the LED frequency encoding table on the calibration board to determine the corresponding feature point ID.

[0198] S280: 3D centroid calculation: Combining the parameters of the binocular camera, the 3D spatial centroid coordinates of each feature point cluster are calculated using the least squares method.

[0199] Based on the same concept, embodiments of this application provide a 3D visual calibration and localization system based on an event camera, which will be described below in conjunction with... Figure 3 The three-dimensional visual calibration and positioning system based on an event camera provided in the embodiments of this application will be described in detail.

[0200] Figure 3 This is a structural block diagram of a three-dimensional visual calibration and positioning system based on an event camera, as shown in an embodiment of this application.

[0201] like Figure 3 As shown, the event camera-based 3D visual calibration and localization system may include:

[0202] Module 310 is used to construct a light source array, which includes multiple independent light sources assigned with unique frequency codes, and to construct a binocular vision system using at least two event cameras, each equipped with a filter that matches the wavelength of the light signal from the light source array.

[0203] The extraction module 320 is used to acquire the light signal event stream of the light source array through the binocular vision system and extract the effective event set in the light signal event stream. The effective event set includes pixel position, light signal frequency and timestamp.

[0204] The construction module 310 is also used to identify the centroid of pixel coordinates of each independent light source by performing cluster analysis on the frequency of light signals and pixel positions, and to establish a mapping relationship between frequency coding and pixel coordinate centroids.

[0205] Arrangement module 330 is used to arrange multiple auxiliary light sources on the surface of the target object. The frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array.

[0206] The extraction module 320 is also used to acquire the target light signal event stream of the target object through the binocular vision system, extract the target effective event set in the target light signal event stream, and determine the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the target effective event set according to the mapping relationship.

[0207] The calculation module 340 is used to correct and match the set of centroids of pixel coordinates based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and to calculate the three-dimensional spatial coordinates of the auxiliary light source by triangulation.

[0208] The calculation module 340 is also used to calculate the pose of the target object based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source through a rigid transformation algorithm to complete the localization. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0209] In one embodiment, the extraction module 320 is specifically used to create a state matrix, the dimension of which is consistent with the resolution of the event camera. Each element of the state matrix records the timestamp of the last event and the current polarity state of the corresponding pixel. The polarity of a new event in the light signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. When the polarity state changes, the frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the last event of the corresponding pixel recorded in the state matrix. Valid events with frequency values ​​within the preset frequency encoding range of the independent light source are filtered, and the pixel position, frequency, and timestamp of the valid events are stored in the valid event set.

[0210] In one embodiment, the construction module 310 is specifically used to cluster events in the effective event set based on light signal frequency and pixel position to generate multiple event clusters; remove event clusters with fewer than a preset threshold number of elements from the multiple event clusters, and retain the effective event clusters; calculate the centroid of the pixel coordinates of the effective event clusters, and verify the matching degree between the average frequency of the effective event clusters and the preset frequency code of the independent light source; and establish a mapping relationship between the centroid coordinates of the effective event clusters whose matching degree meets the preset error range and the frequency code.

[0211] In one embodiment, the construction module 310 is further configured to, before correcting and binocularly matching the set of pixel coordinate centroids based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source using triangulation, move the light source array at multiple angles within the field of view of each event camera, collect the light signal event stream of the light source array in real time, and extract valid recognition results; extract multiple sets of calibration board key point position data according to the mapping relationship between the frequency encoding and pixel coordinate centroids in the valid recognition results; calculate the intrinsic parameter matrix and distortion parameters of the event camera using a planar calibration algorithm based on the multiple sets of calibration board key point position data; and output the intrinsic parameter matrix and distortion parameters of the event camera when the number of valid recognition results reaches a preset threshold and the calibration board key point position data covers the entire field of view of the event camera.

[0212] In one embodiment, the construction module 310 is further configured to move a single frequency-coded light source within the overlapping field of view of the binocular vision system, and simultaneously acquire optical signal event stream data from two event cameras; generate multiple sets of corresponding point pairs by matching the pixel coordinates of the same light source detected by the two event cameras according to the frequency coding; and obtain the extrinsic parameter matrix of the binocular vision system by calculating the rotation matrix and translation vector through essential matrix decomposition based on the corresponding point pairs and the intrinsic parameter matrix.

[0213] In one embodiment, the calculation module 340 is specifically used to perform nonlinear distortion correction on the set of centroids of pixel coordinates using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix, generating corrected pixel coordinates; based on the rotation matrix and translation vector in the extrinsic parameter matrix, perform stereo epipolar correction on the coordinate system of the event camera of the binocular vision system, completing the coplanar and row alignment of the imaging plane of the event camera, forming a binocular stereo correction mapping relationship; according to the binocular stereo correction mapping relationship, perform epipolar constraint matching on the corrected pixel coordinates to obtain the target pixel coordinates, and calculate the horizontal disparity value by determining the matching point pairs of the same auxiliary light source in the images of different event cameras; according to the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, calculate the depth information using the triangulation formula, and convert the depth information and the target pixel to obtain the three-dimensional spatial coordinates.

[0214] In one embodiment, the calculation module 340 is specifically used to perform feature point matching between the three-dimensional spatial coordinates of the auxiliary light source and the distribution information of the auxiliary light source to establish an initial pose correspondence; construct an objective function based on the iterative nearest point algorithm, and optimize the rotation matrix and translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source; when the iterative optimization meets the preset convergence condition, output the optimized rotation matrix and translation vector to obtain the pose of the target object to complete the localization.

[0215] Figure 3 Each module in the system shown has an implementation Figure 1 and Figure 2 The functions of each step in the process and their corresponding technical effects are described in detail here for the sake of brevity.

[0216] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application is shown.

[0217] The electronic device may include a processor 410 and a memory 420 storing computer program instructions.

[0218] Specifically, the processor 410 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0219] Memory 420 may include mass storage for data or instructions. For example, and not limitingly, memory 420 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 420 may include removable or non-removable (or fixed) media. Where appropriate, memory 420 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 420 is non-volatile solid-state memory.

[0220] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0221] The processor 410 reads and executes computer program instructions stored in the memory 420 to implement any of the event camera-based 3D visual calibration and localization methods in the above embodiments.

[0222] In one example, the electronic device may also include a communication interface 430 and a bus 440. Wherein, such as Figure 4 As shown, the processor 410, memory 420, and communication interface 430 are connected via bus 440 and communicate with each other.

[0223] The communication interface 430 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0224] Bus 440 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 440 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0225] The electronic device can execute the event camera-based 3D visual calibration and localization method described in the embodiments of this application, thereby achieving a combination of Figure 1 and Figure 2 The method for 3D visual calibration and localization based on event cameras is described.

[0226] Furthermore, in conjunction with the event-camera-based 3D visual calibration and localization methods described in the above embodiments, this application can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the event-camera-based 3D visual calibration and localization methods described in the above embodiments.

[0227] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0228] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0229] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0230] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0231] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A three-dimensional visual calibration and localization method based on an event camera, characterized in that, include: A light source array is constructed, comprising multiple independent light sources assigned unique frequency codes. A binocular vision system is constructed using at least two event cameras, each equipped with a filter that matches the wavelength of the light signal from the light source array. The binocular vision system acquires the light signal event stream of the light source array and extracts the set of valid events from the light signal event stream. The set of valid events includes pixel position, light signal frequency and timestamp. By performing cluster analysis on the optical signal frequency and the pixel position, the centroid of pixel coordinates for each independent light source is identified, and a mapping relationship between the frequency encoding and the centroid of pixel coordinates is established. Multiple auxiliary light sources are arranged on the surface of the target object, and the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array. The binocular vision system acquires the target light signal event stream of the target object, extracts the target valid event set in the target light signal event stream, and determines the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the target valid event set according to the mapping relationship. Based on the preset intrinsic parameter matrix and distortion parameters of the event camera, and the extrinsic parameter matrix of the binocular vision system, the set of centroids of the pixel coordinates is corrected and binocularly matched, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation. Based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the pose of the target object is calculated using a rigid transformation algorithm to complete the localization. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

2. The method according to claim 1, characterized in that, The extraction of the set of valid events from the optical signal event stream includes: Create a state matrix whose dimensions are consistent with the resolution of the event camera. Each element of the state matrix records the timestamp of the last event and the current polarity state of the corresponding pixel. The polarity of a new event in the optical signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. When the polarity state changes, the frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the previous event of the corresponding pixel recorded in the state matrix. Valid events whose frequency values ​​fall within the preset frequency encoding range of the independent light source are selected, and the pixel position, frequency, and timestamp of the valid events are stored in the valid event set.

3. The method according to claim 1, characterized in that, The step of clustering analysis on the optical signal frequency and the pixel position to identify the centroid of the pixel coordinates of each independent light source and establishing a mapping relationship between the frequency encoding and the centroid of the pixel coordinates includes: The events in the valid event set are clustered based on the optical signal frequency and the pixel position to generate multiple event clusters; Remove event clusters from the plurality of event clusters whose number of elements is less than a preset threshold, and retain the valid event clusters; Calculate the centroid of the pixel coordinates of the effective event cluster, and verify the matching degree between the average frequency of the effective event cluster and the preset frequency code of the independent light source; A mapping relationship is established between the centroid coordinates of the valid event clusters whose matching degree meets the preset error range and the frequency encoding.

4. The method according to claim 1, characterized in that, Before correcting and matching the set of centroids of the pixel coordinates based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and before calculating the three-dimensional spatial coordinates of the auxiliary light source using triangulation, the method includes: The light source array is moved at multiple angles within the field of view of each event camera to collect the light signal event stream of the light source array in real time and extract the valid recognition results. Based on the mapping relationship between the frequency encoding and the centroid of the pixel coordinates in the effective recognition results, multiple sets of calibration board key point position data are extracted; Based on the multiple sets of key point location data of the calibration board, the intrinsic parameter matrix and distortion parameters of the event camera are calculated by a planar calibration algorithm; When the number of valid recognition results reaches a preset threshold and the key point location data of the calibration board covers the entire field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

5. The method according to claim 4, characterized in that, The method further includes: A single frequency-coded light source is moved within the overlapping field of view of the binocular vision system, and optical signal event stream data of the two event cameras are acquired simultaneously. Based on the frequency encoding, the pixel coordinates of the same light source detected by the two event cameras are matched to generate multiple sets of corresponding point pairs; Based on the corresponding point pairs and the intrinsic parameter matrix, the rotation matrix and translation vector are calculated through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system.

6. The method according to claim 1, characterized in that, The process of correcting and binocular matching the set of pixel coordinate centroids based on the intrinsic parameter matrix, the distortion parameters, and the extrinsic parameter matrix, and calculating the three-dimensional spatial coordinates of the auxiliary light source using triangulation, includes: Using the camera focal length, principal point coordinate parameters and distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the set of centroids of pixel coordinates to generate corrected pixel coordinates; Based on the rotation matrix and translation vector in the extrinsic parameter matrix, the coordinate system of the event camera in the binocular vision system is subjected to stereo epipolar correction, and the imaging planes of the event camera are coplanar and aligned, forming a binocular stereo correction mapping relationship. Based on the binocular stereo correction mapping relationship, the target pixel coordinates are obtained by epipolar constraint matching of the corrected pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras. Based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, depth information is calculated using triangulation formulas, and the three-dimensional spatial coordinates are obtained based on the depth information and the target pixel.

7. The method according to claim 1, characterized in that, The step of calculating the pose of the target object using a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete the localization includes: The three-dimensional spatial coordinates of the auxiliary light source are matched with the distribution information of the auxiliary light source using feature points to establish an initial pose correspondence. The objective function is constructed based on the iterative nearest point algorithm. The rotation matrix and translation vector are optimized by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source. When the iterative optimization meets the preset convergence condition, the optimized rotation matrix and translation vector are output to obtain the pose of the target object and complete the localization.

8. A three-dimensional visual calibration and localization system based on an event camera, characterized in that, The system includes: A building module is used to build a light source array, which includes multiple independent light sources assigned with unique frequency codes, and to build a binocular vision system using at least two event cameras, wherein the event cameras are equipped with filters that match the wavelength of the light signals from the light source array. The extraction module is used to acquire the light signal event stream of the light source array through the binocular vision system, and extract the set of valid events in the light signal event stream, wherein the set of valid events includes pixel position, light signal frequency and timestamp; The construction module is also used to identify the centroid of pixel coordinates of each independent light source by performing cluster analysis on the frequency of the light signal and the pixel position, and to establish a mapping relationship between the frequency code and the centroid of pixel coordinates; The arrangement module is used to arrange multiple auxiliary light sources on the surface of the target object, wherein the frequency encoding of the auxiliary light sources is the same as the frequency encoding rule of the light source array; The extraction module is also used to acquire the target light signal event stream of the target object through the binocular vision system, extract the target valid event set in the target light signal event stream, and determine the set of pixel coordinate centroids corresponding to the frequency code of each auxiliary light source in the target valid event set according to the mapping relationship; The calculation module is used to correct and perform binocular matching on the set of centroids of the pixel coordinates based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and to calculate the three-dimensional spatial coordinates of the auxiliary light source by triangulation. The calculation module is also used to calculate the pose of the target object to complete the positioning based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source through a rigid transformation algorithm. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

9. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the event camera-based three-dimensional visual calibration and localization method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the event camera-based three-dimensional visual calibration and localization method as described in any one of claims 1-7.

Images