Three-dimensional vision calibration and positioning method and system based on event camera

Abstract

The invention discloses a three-dimensional vision calibration and positioning method and system based on an event camera, and belongs to the field of positioning. The method comprises the steps of constructing an array containing a plurality of unique frequency coding light sources, constructing a binocular event camera system, collecting light source array light signal event streams, extracting pixel positions, frequencies and timestamp information, determining pixel coordinate centroids of the light sources through frequency and pixel clustering analysis, and establishing frequency and centroid mapping. Auxiliary light sources with the same coding rule are arranged on the surface of a target object, and after a binocular system collects light signals of the auxiliary light sources, the center-of-mass coordinates of the light sources are matched according to the mapping relation. And based on event camera calibration parameters and binocular external parameters, correcting and stereoscopically matching centroid coordinates, and calculating three-dimensional coordinates of the auxiliary light source through triangulation. And in combination with the known geometric topological relation of the auxiliary light source on the surface of the object, the spatial pose of the target object is solved by using a rigid transformation algorithm, and high-precision positioning is completed. According to the embodiment, the calibration precision can be improved in a complex application environment.

Description

Technical Field

[0001] The present application relates to the field of positioning, and in particular to a three-dimensional visual calibration and positioning method and system based on an event camera. Background Art

[0002] Three-dimensional visual calibration and positioning technology is a key foundation for fields such as robotic navigation, autonomous driving, and industrial automation. It acquires the spatial position information of target objects, supporting precise control and environmental perception. Traditional methods rely on visible light cameras and calibration plates for parameter calibration, and localize objects using binocular vision or structured light technologies. However, existing solutions face significant challenges in real-time performance and robustness in dynamic environments or complex lighting conditions.

[0003] In existing technologies, three-dimensional calibration typically uses two-dimensional calibration plates, such as checkerboards or dot arrays, to determine camera parameters through multi-angle image acquisition. Positioning relies on solutions such as binocular stereo matching, structured light projection, or lidar. For example, binocular vision calculates depth information through parallax, but requires high computing resources and is sensitive to texture and lighting. Structured light is easily affected by ambient light, making it difficult to use outdoors. LiDAR, while highly accurate, is expensive and has difficulty detecting transparent objects.

[0004] Therefore, the above method has obvious drawbacks: First, the traditional calibration plate relies on image feature point recognition, which can easily lead to false feature point detection in uneven lighting or complex backgrounds, affecting calibration accuracy. Second, the positioning algorithm based on RGB images has high computational complexity and poor adaptability to highly reflective or transparent objects. Therefore, the existing method suffers from poor calibration accuracy and inability to adapt to complex application environments. Summary of the Invention

[0005] The present application provides a three-dimensional visual calibration and positioning method, system, device and computer storage medium based on an event camera, which can improve calibration accuracy in complex application environments.

[0006] In a first aspect, the present application provides a three-dimensional visual calibration and positioning method based on an event camera, the method comprising:

[0007] Constructing a light source array, the light source array includes multiple independent light sources assigned unique frequency codes, and constructing a binocular vision system using at least two event cameras, wherein the event cameras are equipped with filters that match the wavelength of the light signal of the light source array;

[0008] The binocular vision system collects the light signal event stream of the light source array and extracts the valid event set in the light signal event stream, which includes the pixel position, light signal frequency and time stamp;

[0009] By performing cluster analysis on the light signal frequency and pixel position, the pixel coordinate centroid of each independent light source is identified, and a mapping relationship between frequency encoding and pixel coordinate centroid is established;

[0010] Arrange multiple auxiliary light sources on the surface of the target object, and the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array;

[0011] The target light signal event stream of the target object is collected through the binocular vision system, the target valid event set in the target light signal event stream is extracted, and the pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set is determined according to the mapping relationship;

[0012] Based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation;

[0013] According to the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the position of the target object is calculated through the rigid transformation algorithm to complete the positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0014] In one feasible implementation, extracting a valid event set from an optical signal event stream includes:

[0015] Create a state matrix. The dimension of the state matrix is consistent with the resolution of the event camera. Each element of the state matrix records the last event timestamp and current polarity state of the corresponding pixel.

[0016] The polarity of a new event in the light signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. When the polarity state changes, the frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the previous event of the corresponding pixel recorded in the state matrix.

[0017] Valid events whose frequency values are within the preset frequency coding range of the independent light source are screened, and the pixel position, frequency and time stamp of the valid event are stored in a valid event set.

[0018] In one achievable embodiment, cluster analysis is performed on the optical signal frequency and pixel position to identify the pixel coordinate centroid of each independent light source, and a mapping relationship between the frequency code and the pixel coordinate centroid is established, including:

[0019] Clustering the events in the valid event set based on light signal frequency and pixel position to generate multiple event clusters;

[0020] Remove event clusters whose number of elements is lower than a preset threshold value from multiple event clusters, and retain valid event clusters;

[0021] Calculate the pixel coordinate centroid of the valid event cluster and verify the matching degree between the average frequency of the valid event cluster and the preset frequency encoding of the independent light source;

[0022] A mapping relationship is established between the centroid coordinates of the valid event clusters whose matching degree meets the preset error range and the frequency code.

[0023] In one possible implementation, before correcting the pixel coordinate centroid set and performing binocular matching based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source by triangulation, the method includes:

[0024] The light source array is moved at multiple angles within the field of view of each event camera to collect the light signal event stream of the light source array in real time and extract effective recognition results;

[0025] Extract multiple sets of calibration plate key point position data based on the mapping relationship between the frequency code and the pixel coordinate centroid in the effective recognition results;

[0026] Based on multiple sets of calibration plate key point position data, the intrinsic parameter matrix and distortion parameters of the event camera are calculated using a planar calibration algorithm.

[0027] When the number of valid recognition results reaches the preset threshold and the key point position data of the calibration plate covers the complete field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

[0028] In one possible implementation, the method further includes:

[0029] A single frequency-coded light source is moved within the overlapping fields of view of the binocular vision system to synchronously collect light signal event stream data from two event cameras.

[0030] Match the pixel coordinates of the same light source detected by two event cameras according to frequency coding to generate multiple sets of corresponding point pairs;

[0031] Based on the corresponding point pairs and the intrinsic parameter matrix, the rotation matrix and translation vector are calculated through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system.

[0032] In one feasible implementation, based on the intrinsic parameter matrix, the distortion parameters, and the extrinsic parameter matrix, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation, including:

[0033] Using the camera focal length, principal point coordinate parameters and distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the pixel coordinate centroid set to generate the corrected pixel coordinates;

[0034] Based on the rotation matrix and translation vector in the extrinsic matrix, the coordinate system of the event camera of the binocular vision system is subjected to stereo epipolar correction, so that the imaging planes of the event cameras are coplanar and aligned, thus forming a binocular stereo correction mapping relationship.

[0035] According to the binocular stereo correction mapping relationship, the target pixel coordinates are obtained by performing epipolar constraint matching on the corrected pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras;

[0036] According to the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix and the disparity value, the depth information is calculated by the triangulation formula, and the three-dimensional space coordinates are converted based on the depth information and the target pixel.

[0037] In one feasible implementation, the position of the target object is calculated by a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete the positioning, including:

[0038] Match the three-dimensional spatial coordinates of the auxiliary light source with the distribution information of the auxiliary light source to establish the initial pose correspondence;

[0039] The objective function is constructed based on the iterative closest point algorithm, and the rotation matrix and translation vector are optimized by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source.

[0040] When the iterative optimization meets the preset convergence conditions, the optimized rotation matrix and translation vector are output to obtain the position and posture of the target object to complete positioning.

[0041] In a second aspect, the present application provides a three-dimensional visual calibration and positioning system based on an event camera, the system comprising:

[0042] A construction module is used to construct a light source array, wherein the light source array includes multiple independent light sources assigned with unique frequency codes, and a binocular vision system is constructed by at least two event cameras, wherein the event cameras are equipped with filters that match the wavelength of the light signal of the light source array;

[0043] An extraction module is used to collect the light signal event stream of the light source array through the binocular vision system and extract the valid event set in the light signal event stream, where the valid event set includes pixel position, light signal frequency and timestamp;

[0044] The construction module is further used to identify the pixel coordinate centroid of each independent light source by performing cluster analysis on the light signal frequency and pixel position, and establish a mapping relationship between the frequency code and the pixel coordinate centroid;

[0045] an arrangement module, configured to arrange a plurality of auxiliary light sources on the surface of the target object, wherein the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array;

[0046] The extraction module is further used to collect the target light signal event stream of the target object through the binocular vision system, extract the target valid event set in the target light signal event stream, and determine the pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set according to the mapping relationship;

[0047] A calculation module is used to correct the pixel coordinate centroid set and perform binocular matching based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculate the three-dimensional spatial coordinates of the auxiliary light source through triangulation;

[0048] The calculation module is also used to calculate the position of the target object through a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0049] In a third aspect, the present application provides an electronic device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement a three-dimensional visual calibration and positioning method based on an event camera as in any embodiment of the first aspect.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the event camera-based three-dimensional visual calibration and positioning method as in any one of the embodiments of the first aspect is implemented.

[0051] The present application discloses a 3D visual calibration and positioning method, system, device, and computer storage medium based on an event camera. This technical solution effectively solves the problems of poor calibration accuracy and inability to adapt to complex application environments in traditional technologies by adopting a 3D visual calibration and positioning method based on an event camera. Specifically, this solution utilizes the event camera's keen perception of light and dark changes in the image, combined with an infrared LED stroboscopic calibration plate and markers, to achieve accurate calibration under conditions of uneven lighting or complex backgrounds, avoiding the false detection problem caused by the traditional calibration plate's reliance on image feature point recognition, and significantly improving calibration accuracy. At the same time, by abandoning the complex RGB image processing algorithm, the amount of calculation is greatly reduced, the real-time performance is improved, and excellent recognition accuracy and robustness are demonstrated for highly reflective or transparent objects, effectively overcoming the limitations of existing technologies in complex application environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a flow chart of a three-dimensional visual calibration and positioning method based on an event camera provided by an embodiment of the present application;

[0054] Figure 2 This is a flow chart of a three-dimensional positioning algorithm based on infrared stroboscopic features of an event camera provided by an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of the structure of a three-dimensional visual calibration and positioning system based on an event camera provided by an embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0058] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0059] To solve the problems of the prior art, the present invention provides a method, system, device, and computer storage medium for three-dimensional visual calibration and positioning based on an event camera. The following first introduces the three-dimensional visual calibration and positioning method based on an event camera provided in the present invention.

[0060] Figure 1 FIG1 shows a flow chart of a three-dimensional visual calibration and positioning method based on an event camera provided by an embodiment of the present application. Figure 1 As shown, the method includes steps S110 to S170.

[0061] S110: Construct a light source array, where the light source array includes multiple independent light sources assigned unique frequency codes, and construct a binocular vision system through at least two event cameras. The event cameras are equipped with filters that match the wavelength of light signals from the light source array.

[0062] A light source array is a calibration device, a stroboscope calibration board based on infrared LEDs. Constructing a light source array is a crucial step in 3D vision calibration and positioning systems. A light source array consists of multiple independent light sources arranged in a regular pattern. These light sources emit light signals encoded with a specific frequency, providing the foundation for subsequent event camera acquisition and feature point recognition.

[0063] The calibration plate is fabricated as follows: the light source array typically consists of an array of infrared LEDs (light-emitting diodes) arranged in m rows and n columns. Each infrared LED is assigned a unique ID and sequentially coded according to that ID. This coding facilitates accurate identification and location of each light source during subsequent processing.

[0064] Infrared LEDs with a 940nm wavelength are typically selected for use with event cameras and filters. This wavelength offers low background noise and is easily filtered by specific filters, thereby improving signal detection accuracy. The infrared LEDs are mounted on a flat plate to ensure stability and consistency across the light source array. The plate selection should consider factors such as flatness, heat resistance, and mechanical strength to adapt to varying operating environments.

[0065] In order to accurately identify and distinguish each individual light source in an array of light sources, they need to be frequency-encoded. Frequency encoding is an effective identification method that assigns a unique frequency to each light source, allowing each light source to be accurately identified in a stream of optical signal events.

[0066] A 555 timer configured in astable mode generates a PWM (pulse-width modulation) signal to control the flickering of each infrared LED. The PWM signal can be adjusted to control the LED's brightness and flickering frequency by adjusting the duty cycle. By designing different control circuits, each infrared LED can be made to flicker at a different frequency.

[0067] The frequency of the lamp with ID 1 is set to 1kHz. Starting with lamp ID 2, the frequency increases by 0.1kHz for each lamp. For example, the frequency of lamp ID 2 is 1.1kHz, the frequency of lamp ID 3 is 1.2kHz, and so on. This frequency encoding method is simple and effective, ensuring that each light source is uniquely identified. The frequency selection should take into account the response speed and data processing capabilities of the event camera. Excessively high frequencies may prevent the event camera from accurately detecting events, while excessively low frequencies may reduce the system's real-time performance. The control circuit is integrated onto a PCB (printed circuit board) to control the infrared LED array. The PCB design should take into account factors such as circuit stability, reliability, and heat dissipation to ensure long-term stable operation of the light source array.

[0068] An event camera is one of the core sensors in a 3D visual calibration and positioning system. It detects changes in brightness and darkness in an image and outputs data as an event stream. Compared to traditional cameras, event cameras offer advantages such as higher dynamic range, lower latency, and lower power consumption.

[0069] An event camera is a biomimetic camera that can detect changes in brightness within an image. Its chip, unlike traditional CCD and CMOS, utilizes a completely new architecture. Each pixel operates asynchronously, independently monitoring changes in brightness. When the change exceeds a certain threshold, it outputs an "event" in the format (x, y, p, t), where x and y represent the pixel location where the event occurred, p is the polarity, taking a value of 1 or 0, indicating whether the point brightened (an on event) or dimmed (an off event), and t is a microsecond timestamp. Therefore, its raw data is an event stream composed of countless events, rather than image frames. Since it only outputs the portions that have changed, the data volume is greatly reduced, making it easier to transmit and process, laying the foundation for real-time detection.

[0070] An event camera is selected as the system's sensor. It detects changes in brightness and darkness in the image. Each pixel operates asynchronously, generating an event when the change exceeds a set threshold. This event contains information such as pixel location, polarity, and timestamp, providing rich data for subsequent feature point recognition and positioning.

[0071] When selecting an event camera, you should consider parameters such as resolution, frame rate, dynamic range, and sensitivity. A camera with high resolution can provide more detailed spatial information, a camera with high frame rate can capture faster motion changes, and a camera with high dynamic range and sensitivity can maintain stable performance under complex lighting conditions.

[0072] A binocular vision system consists of at least two event cameras. By simulating the human eye's binocular vision principle, it can obtain depth information about objects, thereby achieving 3D visual calibration and positioning. Two event cameras form a binocular stereo vision system. Each camera captures the same scene from different angles. By comparing the differences in the images captured by the two cameras, the depth of the object can be calculated. The cameras are installed so that their fields of view cover the area to be measured and there is sufficient overlap. This overlap is the basis for the binocular vision system to perform stereo matching and depth calculation.

[0073] Baseline distance and camera angle are key parameters for binocular vision systems. The baseline distance is set to 1 / 4 the distance to the object being measured for higher accuracy. The camera angle is determined based on the actual object being measured. Generally speaking, a larger angle improves measurement accuracy, but the closer the measurement distance, the smaller the measurement range. Therefore, in practical applications, a trade-off and selection must be made based on the specific scenario.

[0074] The filter is an optical component mounted on the event camera lens. It filters out light that doesn't match the wavelength of the light signal from the light source array, thereby reducing background noise and improving the accuracy of light signal event detection. A 940nm bandpass filter is selected. This filter allows light with a wavelength of 940nm to pass while blocking light of other wavelengths. This matches the wavelength of the infrared LEDs in the light source array and effectively filters out background noise. When the filter is installed on the camera lens, only light near the 940nm wavelength is allowed to pass. This reduces background and noise interference, ensuring that the camera captures a majority of valid data. Increasing the amount of valid data reduces the total data volume, easing data processing complexity, and improving the system's real-time performance and accuracy.

[0075] The specific implementation process is as follows: First, a light source array is constructed. This involves selecting multiple independent light sources and assigning each a unique frequency code. These light sources are arranged according to a specific pattern to form a light source array. The design of the light source array must take into account factors such as the distribution of the light sources, the assignment of frequency codes, and the luminous intensity of the light sources to ensure that each light source can be accurately identified and located during subsequent processing.

[0076] Secondly, a binocular vision system is constructed using at least two event cameras. The selection of event cameras requires consideration of parameters such as resolution, frame rate, and dynamic range to meet the requirements of 3D vision calibration and positioning. The binocular vision system must ensure that the positions and angles of the two event cameras cover the motion range of the light source array and the target object, thereby obtaining sufficient information for 3D reconstruction and positioning.

[0077] Finally, the event camera is equipped with a filter that matches the wavelength of the light signal from the light source array. The filter must be selected to match the wavelength of the light signal emitted by the light source array to effectively filter out background noise and improve the detection accuracy of light signal events. The filter must be installed to ensure that it accurately covers the event camera lens to avoid light leakage and interference.

[0078] S120: Collecting a light signal event stream of the light source array through a binocular vision system, and extracting a valid event set in the light signal event stream, where the valid event set includes a pixel position, a light signal frequency, and a timestamp.

[0079] Specifically, extracting a valid event set from the light signal event stream in step S120 may include: creating a state matrix, wherein the dimension of the state matrix is consistent with the resolution of the event camera, and each element of the state matrix records the last event timestamp and current polarity state of the corresponding pixel; comparing the polarity of a new event in the light signal event stream with the current polarity state of the corresponding pixel in the state matrix, and when the polarity state changes, calculating a frequency value based on the difference between the timestamp of the new event and the timestamp of the last event of the corresponding pixel recorded in the state matrix; screening valid events whose frequency values are within the preset frequency coding range of the independent light source, and storing the pixel position, frequency, and timestamp of the valid event in the valid event set.

[0080] The light signal event stream refers to the series of events triggered by the light signals emitted by the light source array, captured by the event camera. An event camera is a biomimetic camera that detects changes in brightness within an image. Each pixel operates asynchronously, outputting an event when the brightness change exceeds a certain threshold. These events contain information such as pixel location, polarity, and timestamp, forming the light signal event stream. A valid event set is a set of events extracted from the light signal event stream that meet specific conditions, such as having a frequency value within the preset frequency encoding range of an independent light source. This valid event set, which contains information such as pixel location, light signal frequency, and timestamp, serves as the basis for subsequent processing and analysis. The state matrix is a matrix used to record the state of each pixel. Its dimensions match the resolution of the event camera. Each element of the state matrix records the timestamp of the previous event and the current polarity state of the corresponding pixel. The state matrix allows for real-time tracking and updating of each pixel's state, thereby accurately extracting the valid event set.

[0081] The specific implementation process is as follows: First, a binocular vision system collects the light signal event stream from the light source array. The binocular vision system consists of at least two event cameras, each equipped with filters that match the wavelength of the light signal from the light source array to ensure accurate capture of the light signal emitted by the light source array. When individual light sources in the light source array flicker, the event camera detects the change in brightness in the image and outputs the corresponding event stream data.

[0082] Next, a valid event set is extracted from the light signal event stream. This process includes the following substeps: Create a state matrix: A new state matrix S is created with dimensions consistent with the resolution of the event camera. Each element of the state matrix records the timestamp of the last event and the current polarity state of the corresponding pixel. Initially, the polarity state of all pixels can be set to a default value, such as "no change."

[0083] Real-time state matrix update: Whenever a new event occurs, the polarity of the new event in the light signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. If the polarity state changes (for example, from "no change" to "brighter" or "darker"), a frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the previous event for the corresponding pixel recorded in the state matrix. This frequency value reflects the frequency of change of the light signal at that pixel location.

[0084] Filter valid events: Based on the calculated frequency values, select valid events that fall within the preset frequency encoding range of the individual light sources. These valid events correspond to events caused by the flickering of individual light sources in the light source array. Information such as the pixel location, frequency, and timestamp of the valid events is stored in a valid event set.

[0085] Maintaining the valid event set: To ensure the timeliness and accuracy of the valid event set, it is necessary to update and maintain it in real time. For example, you can set a time window to only retain valid events within the latest period of time and eliminate events that are too old.

[0086] Through these sub-steps, a valid event set can be accurately extracted from the light signal event stream collected by the binocular vision system. This valid event set contains key information about the individual light sources in the light source array, providing an important data foundation for subsequent steps such as feature point recognition, 3D calibration, and positioning.

[0087] S130: performing cluster analysis on the light signal frequency and pixel position to identify the pixel coordinate centroid of each independent light source and establish a mapping relationship between the frequency code and the pixel coordinate centroid.

[0088] Specifically, step S130: by performing cluster analysis on the light signal frequency and pixel position, identifying the pixel coordinate centroid of each independent light source, and establishing a mapping relationship between the frequency code and the pixel coordinate centroid, including: clustering the events in the valid event set based on the light signal frequency and pixel position to generate multiple event clusters; removing event clusters whose number of elements is lower than a preset number threshold in the multiple event clusters, and retaining the valid event clusters; calculating the pixel coordinate centroid of the valid event cluster, and verifying the matching degree between the average frequency of the valid event cluster and the preset frequency code of the independent light source; establishing a mapping relationship between the centroid coordinates of the valid event cluster whose matching degree meets the preset error range and the frequency code.

[0089] Cluster analysis is a statistical method used to group objects in a dataset based on similarity. In this solution, cluster analysis is applied to the set of valid events in the light signal event stream. By analyzing the light signal frequency and pixel location of the events, similar events are grouped together, thereby identifying the pixel coordinate centroid of each independent light source.

[0090] Event clusters are the result of cluster analysis and represent a group of events with similar characteristics. In this scheme, event clusters are collections of events with similar light signal frequencies and pixel locations within the light signal event stream. By analyzing event clusters, the pixel coordinate centroid of each individual light source can be identified.

[0091] The pixel coordinate centroid is the average of all event pixel positions in an event cluster and represents the center of the event cluster in the image. In this scheme, the pixel coordinate centroid is used to establish a mapping relationship between frequency encoding and pixel coordinates, providing a basis for subsequent 3D spatial coordinate calculations.

[0092] A mapping relationship refers to the one-to-one correspondence between frequency codes and pixel coordinate centroids. In this solution, after identifying the pixel coordinate centroid of each independent light source through cluster analysis, it is necessary to establish a mapping relationship between this centroid and the light source's frequency code. This allows for rapid location of the corresponding pixel coordinate centroid based on the frequency code in subsequent processing.

[0093] The specific implementation process is as follows: First, cluster analysis is performed on the events in the valid event set. This step aims to group events with similar light signal frequencies and pixel locations together, thereby identifying all events generated by each independent light source in the image. Cluster analysis can be performed using a variety of algorithms, such as K-means and DBSCAN. The specific algorithm should be adjusted based on the actual data and scenario.

[0094] During cluster analysis, multiple event clusters are generated. Each event cluster represents all events generated by a single light source in the image. However, not all event clusters are valid. Some clusters may contain only a small number of events, possibly due to noise or interference. Therefore, it is necessary to filter the event clusters, removing those with fewer elements than a preset threshold and retaining only valid clusters.

[0095] Next, the pixel coordinate centroid of the valid event cluster is calculated. The pixel coordinate centroid is the average of all event pixel positions in the event cluster and represents the center of the event cluster in the image. By calculating the pixel coordinate centroid, the position of each independent light source in the image can be accurately located.

[0096] Next, the average frequency of the valid event cluster is verified to match the preset frequency code of the individual light source. Since each individual light source is assigned a unique frequency code, the average frequency of the valid event cluster can be calculated and compared with its preset frequency code to verify whether the event cluster is indeed generated by a specific individual light source. If the match falls within the preset error range, the event cluster is considered valid, and a mapping relationship is established between its centroid coordinates and the frequency code.

[0097] Finally, a mapping relationship is established between the centroid coordinates of valid event clusters whose matching degree falls within a preset error range and the frequency code. This step is the core objective of step S130. By establishing a mapping relationship between the frequency code and the pixel coordinate centroid, it provides a foundation for subsequent three-dimensional space coordinate calculations. In subsequent processing, when it is necessary to locate an independent light source, simply find the corresponding pixel coordinate centroid based on its frequency code to quickly locate the light source in the image.

[0098] S140: Arrange a plurality of auxiliary light sources on the surface of the target object, wherein the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array.

[0099] Auxiliary light sources are multiple light sources placed on the surface of a target object. These light sources share the same frequency coding rules as the individual light sources in the light source array. The purpose of these auxiliary light sources is to create feature points on the surface of the target object, which are then captured and located by the binocular vision system.

[0100] The frequency coding rule refers to the rule for assigning a unique frequency code to each independent light source in the light source array. In step S140, the frequency coding of the auxiliary light source needs to be the same as the frequency coding rule of the light source array to ensure that each auxiliary light source can be accurately identified and located in the subsequent processing.

[0101] The specific implementation process is as follows: First, auxiliary light sources must be placed at appropriate locations on the target object's surface. These locations should fully reflect the target's shape and posture while avoiding mutual interference and occlusion. The number and placement of auxiliary light sources should be determined based on the target's complexity and positioning accuracy requirements.

[0102] When arranging auxiliary light sources, it is necessary to ensure that the frequency coding of each auxiliary light source follows the same rules as the individual light sources in the light source array. This means that each auxiliary light source should also be assigned a unique frequency code, and this code should correspond to the code of the individual light source at the corresponding position in the light source array. By maintaining the consistency of the frequency coding rules, it can be ensured that the binocular vision system can accurately identify and locate each auxiliary light source during subsequent processing.

[0103] To achieve this goal, the auxiliary light source can be constructed using the same light source and control circuitry as the light array. Specifically, the same infrared LEDs as those in the light array can be used and assigned the same frequency coding as the corresponding independent light sources in the array. Furthermore, the control circuitry of the auxiliary light source must be synchronized with that of the light array to ensure consistent and accurate frequency coding.

[0104] S150: Collect the target light signal event stream of the target object through the binocular vision system, extract the target valid event set in the target light signal event stream, and determine the pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set according to the mapping relationship.

[0105] The target light signal event stream refers to a series of events triggered by light signals emitted by auxiliary light sources placed on the surface of the target object, collected by the binocular vision system. These events contain information such as pixel location, light signal frequency, and timestamp, forming the target light signal event stream.

[0106] The target valid event set is a set of events extracted from the target light signal event stream that meet specific conditions, such as a frequency value within the preset frequency encoding range of the auxiliary light source. The target valid event set contains information such as pixel location, light signal frequency, and timestamp, and serves as the basis for subsequent processing and analysis.

[0107] The pixel coordinate centroid set is the set of pixel coordinate centroids of each auxiliary light source in the image, identified through cluster analysis of events in the target valid event set. These pixel coordinate centroids represent the position of each auxiliary light source in the image and are key information for subsequent 3D coordinate calculations and target object positioning.

[0108] The specific implementation process is as follows: First, a binocular vision system collects the target object's target light signal event stream. The binocular vision system consists of at least two event cameras, each equipped with filters that match the wavelength of the light signal from the light source array. This ensures that they can accurately capture the light signals from the auxiliary light sources placed on the target object's surface. When the auxiliary light sources flicker, the event cameras detect the changes in brightness in the image and output the corresponding event stream data.

[0109] Next, a target valid event set is extracted from the target optical signal event stream. This process is similar to the process of extracting a valid event set in step S120, including steps such as creating a state matrix, updating the state matrix in real time, and screening valid events. Through these steps, a target valid event set that meets specific conditions can be accurately extracted from the target optical signal event stream.

[0110] Finally, the set of pixel coordinate centroids corresponding to the frequency codes of each auxiliary light source in the target valid event set is determined based on the mapping relationship. This step utilizes the mapping relationship between frequency codes and pixel coordinate centroids established in step S130. Through table lookup or calculation, the corresponding pixel coordinate centroid can be quickly located based on the frequency code of each event in the target valid event set. These pixel coordinate centroids constitute the pixel coordinate centroid set, providing key information for subsequent three-dimensional space coordinate calculation and target object positioning.

[0111] S160: Based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation.

[0112] Specifically, step S160: based on the intrinsic parameter matrix, the distortion parameters and the extrinsic parameter matrix, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional space coordinates of the auxiliary light source are calculated by triangulation, including:

[0113] The camera focal length, principal point coordinate parameters and distortion parameters in the intrinsic parameter matrix are used to perform nonlinear distortion correction on the pixel coordinate centroid set to generate corrected pixel coordinates; based on the rotation matrix and translation vector in the extrinsic parameter matrix, the coordinate system of the event camera of the binocular vision system is stereo epipolar corrected to complete the coplanarity and row alignment of the imaging planes of the event camera, forming a binocular stereo correction mapping relationship; according to the binocular stereo correction mapping relationship, the corrected pixel coordinates are subjected to epipolar constraint matching to obtain the target pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras; according to the focal length parameters in the intrinsic parameter matrix, the baseline distance parameters and disparity values in the extrinsic parameter matrix, the depth information is calculated by the triangulation formula, and based on the depth information and the target pixel, the three-dimensional space coordinates are converted.

[0114] Nonlinear distortion correction refers to the process of correcting the pixel coordinate centroid using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix. Images captured by event cameras are distorted due to factors such as lens manufacturing and installation errors. The goal of nonlinear distortion correction is to eliminate this distortion so that the corrected pixel coordinates more accurately reflect the actual position of the target object.

[0115] Stereo epipolar correction refers to the process of correcting the event camera coordinate system of a binocular vision system based on the rotation matrix and translation vector in the extrinsic matrix. The goal of stereo epipolar correction is to make the imaging planes of the two cameras coplanar and aligned, thereby simplifying the stereo matching process and improving matching accuracy and efficiency.

[0116] Epipolar-constrained matching is the process of matching corrected pixel coordinates based on the binocular stereo mapping. Epipolar-constrained matching ensures that matching point pairs from different event camera images for the same auxiliary light source lie on the same epipolar line, thereby improving matching accuracy and robustness.

[0117] Triangulation is a method that calculates the depth of an auxiliary light source based on the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the parallax value, and converts it into three-dimensional spatial coordinates. Triangulation is a commonly used depth calculation method in binocular vision systems. It exploits the geometric relationship between parallax and depth to accurately calculate the three-dimensional spatial coordinates of the target object.

[0118] The specific implementation process is as follows: First, nonlinear distortion correction is performed on the pixel coordinate centroid set using the camera focal length, principal point coordinate parameters, and distortion parameters in the intrinsic parameter matrix. Due to lens distortion, directly using uncorrected pixel coordinates for 3D coordinate calculations introduces errors. Therefore, nonlinear distortion correction is required to eliminate these errors and generate corrected pixel coordinates.

[0119] Next, epipolar correction is performed on the binocular vision system's event camera coordinate system based on the rotation matrix and translation vector in the extrinsic matrix. The goal of epipolar correction is to make the imaging planes of the two cameras coplanar and aligned, thereby simplifying the stereo matching process. Epipolar correction yields a binocular stereo mapping, providing the foundation for subsequent epipolar-constrained matching.

[0120] Epipolar-constrained matching is then performed on the corrected pixel coordinates based on the binocular stereo mapping. This ensures that matching point pairs for the same auxiliary light source in different event camera images lie on the same epipolar line, improving matching accuracy and robustness. This method can be used to obtain matching point pairs for the same auxiliary light source in different event camera images and calculate horizontal disparity.

[0121] Finally, using the focal length parameter in the intrinsic matrix, the baseline distance parameter in the extrinsic matrix, and the parallax value, depth information is calculated using a triangulation formula. Based on this depth information and the target pixel coordinates, the 3D coordinates of the auxiliary light source are converted. Triangulation exploits the geometric relationship between parallax and depth to accurately calculate the 3D coordinates of the target object.

[0122] In one embodiment, to further improve the accuracy and robustness of the calculation of three-dimensional spatial coordinates, multi-view geometric constraints and global optimization methods can be used. Specifically, multiple event cameras can be used to capture the target object from different perspectives to obtain a set of pixel coordinate centroids under multiple perspectives. These pixel coordinate centroid sets are then jointly matched and optimized using multi-view geometric constraints to eliminate mismatches and noise interference. Finally, a global optimization method is used to jointly optimize all matching point pairs to obtain more accurate and robust three-dimensional spatial coordinates. This method can further improve the accuracy and robustness of event camera-based three-dimensional visual calibration and positioning methods.

[0123] S170: Calculate the position of the target object by a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete the positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0124] Specifically, step S170: according to the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the position and posture of the target object are calculated by a rigid transformation algorithm to complete positioning, including: matching the three-dimensional spatial coordinates of the auxiliary light source with the distribution information of the auxiliary light source for feature points to establish an initial position correspondence; constructing an objective function based on an iterative nearest point algorithm, and optimizing the rotation matrix and translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source; when the iterative optimization meets the preset convergence conditions, outputting the optimized rotation matrix and translation vector to obtain the position and posture of the target object to complete positioning.

[0125] A rigid transformation algorithm involves only rotating and translating an object in space without causing deformation. Rigid transformation algorithms exploit this property to determine the object's position in space by calculating a rotation matrix and translation vector. In this solution, a rigid transformation algorithm is used to calculate the target object's position based on the 3D spatial coordinates and distribution information of the auxiliary light source.

[0126] Feature point matching refers to unique and repeatable points in an image, such as corners and edges. Feature point matching involves matching feature points in different images to establish a spatial relationship between them. In this solution, feature point matching is used to establish a correspondence between the auxiliary light source and the target object surface in three-dimensional space.

[0127] The Iterative Closest Point (ICP) algorithm is an algorithm used for 3D point cloud registration. It iteratively searches for the closest point pair between two point clouds and calculates the rotation matrix and translation vector to minimize the spatial distance residual between the two point clouds. In this solution, the ICP algorithm is used to optimize the rotation matrix and translation vector to improve the accuracy of the calculated target object pose.

[0128] The spatial distance residual refers to the difference between the actual spatial distance between two points and the calculated spatial distance. In this scheme, the spatial distance residual is the objective function for optimizing the rotation matrix and translation vector. By minimizing the spatial distance residual, a more accurate rotation matrix and translation vector can be obtained.

[0129] The specific implementation process is as follows: First, feature point matching is performed between the auxiliary light source's 3D spatial coordinates and its distribution information. This information includes the geometric topology and relative position of the auxiliary light source on the target object's surface, which can be obtained through pre-measurement or design. The purpose of feature point matching is to establish a correspondence between the auxiliary light source and the target object's surface in 3D space, providing a foundation for subsequent pose calculations. Feature point matching can determine the initial pose of the auxiliary light source on the target object's surface.

[0130] Next, the objective function is constructed based on the iterative closest point algorithm. The iterative closest point algorithm is an algorithm used to optimize the rotation matrix and translation vector. It optimizes the rotation matrix and translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the auxiliary light source distribution information. In this solution, the objective function is constructed as the sum of the spatial distance residuals between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the auxiliary light source distribution information. The specific formula is as follows:

[0131]

[0132] Among them, E is the spatial distance residual, R is the rotation matrix, T is the translation vector, Pi is the three-dimensional spatial coordinate of the auxiliary light source, Qi is the corresponding coordinate in the auxiliary light source distribution information, and n is the number of auxiliary light sources.

[0133] The rotation matrix and translation vector are then optimized by minimizing the spatial distance residual. This step is the core of the iterative closest point algorithm, which continuously updates the rotation matrix and translation vector in an iterative manner, gradually reducing the spatial distance residual. In each iteration, the algorithm calculates the spatial distance residual under the current rotation matrix and translation vector and updates the rotation matrix and translation vector based on the magnitude and direction of the residual. This process continues until the preset convergence conditions are met.

[0134] Finally, when the iterative optimization meets the preset convergence criteria, the optimized rotation matrix and translation vector are output, and the pose of the target object is obtained to complete the positioning. The preset convergence criteria can include the maximum number of iterations, the spatial distance residual being less than a certain threshold, or the change in the rotation matrix and translation vector being less than a certain threshold. When any of these conditions are met, the iterative optimization is considered to have converged, and the rotation matrix and translation vector at this time are the pose of the target object. This process allows for precise positioning of the target object.

[0135] This embodiment effectively solves the problems of poor calibration accuracy and inability to adapt to complex application environments in traditional technologies by adopting a 3D vision calibration and positioning method based on an event camera. Specifically, this solution utilizes the event camera's keen perception of changes in light and dark in the image, combined with an infrared LED stroboscopic calibration plate and markers, to achieve accurate calibration under conditions of uneven lighting or complex backgrounds, avoiding the false detection problem caused by the traditional calibration plate's reliance on image feature point recognition, and significantly improving calibration accuracy. At the same time, by abandoning the complex RGB image processing algorithm, the amount of calculation is greatly reduced, the real-time performance is improved, and the recognition accuracy and robustness of highly reflective or transparent objects are demonstrated, effectively overcoming the limitations of existing technologies in complex application environments.

[0136] In one feasible embodiment, before step S160: correcting and binocular matching the pixel coordinate centroid set based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source by triangulation, the method includes:

[0137] The light source array is moved at multiple angles within the field of view of each event camera, and the light signal event stream of the light source array is collected in real time to extract valid recognition results. Based on the mapping relationship between the frequency code and the pixel coordinate centroid in the valid recognition results, multiple sets of calibration plate key point position data are extracted. Based on the multiple sets of calibration plate key point position data, the intrinsic parameter matrix and distortion parameters of the event camera are calculated using a plane calibration algorithm. When the number of valid recognition results reaches a preset threshold and the calibration plate key point position data covers the entire field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

[0138] An event camera is a biomimetic camera capable of detecting changes in brightness within an image. Each pixel independently monitors the brightness of its own light source and, when the brightness change exceeds a certain threshold, outputs an event containing its location, polarity, and timestamp. This feature gives event cameras significant advantages in high-speed motion and low-light environments. In this embodiment, the light source array refers to a strobe calibration board based on infrared LEDs. The infrared LEDs are arranged in a specific array and can strobe, with each LED having a unique strobe frequency, which is used for subsequent identification and positioning. The light signal event stream is a data stream consisting of events generated by brightness changes captured by the event camera. These events contain the LED strobe location (x, y coordinates), polarity (brightening or dimming), and timestamp, serving as essential data for subsequent processing. Frequency encoding means that each infrared LED has a unique strobe frequency, which is used as an encoding to identify its position in the array. By decoding this frequency information, the identity and location of the LED can be determined.

[0139] The pixel centroid is the pixel coordinate representing the center of an event cluster, obtained through a clustering algorithm. In the event stream captured by the event camera, events generated by the same LED strobe light form an event cluster. By calculating the centroid of this event cluster, the precise location of the LED in the image can be determined.

[0140] The intrinsic parameter matrix and distortion parameters refer to the camera's internal parameters and parameters caused by lens distortion. The intrinsic parameter matrix includes internal parameters such as the camera's focal length and principal point, while the distortion parameters describe lens distortion. These parameters are the basis for subsequent tasks such as image correction and 3D positioning.

[0141] The detailed process is as follows: First, a stroboscopic calibration board (light source array) based on infrared LEDs is moved at multiple angles within the field of view of each event camera. This step ensures that all LED points on the board are captured from different camera viewing angles, thereby obtaining comprehensive calibration data. This multi-angle movement ensures that the entire camera field of view is covered, while accounting for distortion and perspective changes at different viewing angles.

[0142] Next, the event camera captures the brightness and darkness changes generated by the infrared LED strobe on the calibration board in real time, generating a light signal event stream. These events contain the position, polarity, and timestamp information of the LED strobe, providing essential data for subsequent processing. The characteristics of the event camera enable it to capture clear event streams even in high-speed motion and low-light environments.

[0143] Then, based on the frequency information and pixel coordinates in the event stream, a specific algorithm (such as a clustering algorithm) is used to extract valid recognition results. A valid recognition result is a set of events that successfully identify and match LED points with the predetermined frequency code. This step can filter out events belonging to the LED points on the calibration board, providing accurate data for subsequent processing.

[0144] Next, based on the mapping relationship between the frequency codes and pixel coordinate centroids in the valid recognition results, multiple sets of calibration plate keypoint location data are extracted. Each keypoint location data includes the pixel coordinates of that point in the image and the corresponding frequency code. This keypoint location data is the basis for the subsequent calculation of the camera intrinsic parameter matrix and distortion parameters.

[0145] Then, based on multiple sets of keypoint position data from the calibration plate, a planar calibration algorithm (such as Zhang's calibration method) is used to calculate the intrinsic parameter matrix and distortion parameters of the event camera. The intrinsic parameter matrix includes internal parameters such as the camera's focal length and principal point, while the distortion parameters describe the lens distortion. Using this planar calibration algorithm, we can obtain precise intrinsic parameters and distortion parameters for the camera, providing an accurate foundation for subsequent tasks such as image correction and 3D positioning.

[0146] Finally, when the number of valid recognition results reaches a preset threshold (e.g., 20 or more), and the keypoint position data on the calibration plate covers the entire field of view of the event camera, the calculated intrinsic parameter matrix and distortion parameters are output. These parameters are used for subsequent tasks such as image correction and 3D positioning to ensure the accuracy and stability of the system.

[0147] In a feasible embodiment, the method also includes: moving a single frequency-coded light source within the overlapping field of view of the binocular vision system, and synchronously collecting light signal event stream data of the two event cameras; matching the pixel coordinates of the same light source detected by the two event cameras according to the frequency coding to generate multiple groups of corresponding point pairs; based on the corresponding point pairs and the intrinsic parameter matrix, calculating the rotation matrix and translation vector through essential matrix decomposition to obtain the external parameter matrix of the binocular vision system.

[0148] In this embodiment, the frequency-coded light source refers to an infrared LED light source that can emit light at a specific frequency. Each light source has a unique strobe frequency, and this frequency information is used as a code to identify the light source. By detecting this frequency information, different light sources can be distinguished. A binocular vision system refers to a system consisting of two cameras. The two cameras capture the same object, calculate the parallax, and calculate the object's 3D information based on triangulation. The binocular vision system can obtain depth information of the object and is the basis for achieving three-dimensional visual positioning.

[0149] In a binocular vision system, the intrinsic matrix is a 3x3 matrix used to describe the relative position (rotation and translation) between two cameras and the geometric structure of the scene. By decomposing the essential matrix, the rotation matrix and translation vector can be obtained, thereby determining the relative position relationship between the two cameras. The rotation matrix describes the rotation relationship between the two cameras in three-dimensional space. Through the rotation matrix, a point in one camera coordinate system can be transformed into the coordinate system of another camera. The translation vector describes the translation relationship between the two cameras in three-dimensional space. The translation vector represents the displacement between the origins of the two camera coordinate systems. In a binocular vision system, the extrinsic parameter matrix is composed of the rotation matrix and translation vector, which describes the relative position between the two cameras. The extrinsic parameter matrix is one of the key parameters for achieving three-dimensional visual positioning.

[0150] The process involves the following steps: First, two event cameras are installed at predetermined positions and angles, ensuring that their fields of view overlap. This is the foundation for binocular visual positioning. Only when the two cameras' fields of view overlap can they capture images of the same object from different perspectives, allowing for the calculation of parallax and depth information.

[0151] Next, the two cameras were activated to synchronously collect the light signal event stream data. Because the event cameras only output the changing portion, the data volume is relatively small, making it easier to process in real time. Synchronously collecting event stream data from the two cameras ensures temporal synchronization between the two cameras, providing an accurate foundation for subsequent data processing.

[0152] Then, a frequency-coded infrared LED light source is moved within the overlapping fields of view. This light source emits light of a specific frequency. The two cameras each detect the light source's event stream data in real time and identify the source based on the frequency coding. Frequency coding ensures that both cameras detect the same light source, providing a basis for subsequent data matching.

[0153] Next, by comparing the event stream data detected by the two cameras, the pixel coordinates of the same light source in both cameras are matched, generating multiple sets of corresponding point pairs. These corresponding point pairs serve as the basis for the subsequent calculation of the essential matrix, representing the positional relationship of the same light source from different perspectives in the two cameras.

[0154] Finally, based on the corresponding point pairs and the intrinsic parameter matrix, the rotation matrix and translation vector are calculated through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system. Given the intrinsic parameter matrices of the two cameras (obtained through the previous calibration process), the essential matrix can be calculated using the corresponding point pairs and the intrinsic parameter matrix. Singular value decomposition (SVD) is performed on the essential matrix to obtain candidate solutions for the rotation matrix and translation vector. Triangulation is used to determine whether the depth of the point reprojected to the camera is positive, and the set of solutions with positive depth is retained to obtain the final rotation matrix R and translation vector t. The rotation matrix R and translation vector t together constitute the extrinsic parameter matrix of the binocular vision system, which describes the relative position between the two cameras.

[0155] In one implementation, this embodiment provides a complete 3D visual calibration and positioning method based on an event camera, the details of which are as follows:

[0156] This method realizes 3D vision calibration and positioning based on event cameras, specifically involving a calibration method and device, and a positioning method based on key point recognition.

[0157] First, the calibration board is made. The calibration device mentioned above refers to a stroboscopic calibration board based on infrared LEDs. The specific production method is as follows:

[0158] Step 1: Arrange (m*n) infrared LEDs in an array of m rows and n columns on a flat plate. Number them sequentially, starting with 1, and assign each LED a unique ID. The infrared LED wavelength is 940nm.

[0159] Step 2: Design a control circuit to achieve a strobe-like pattern on the infrared LEDs, with controllable frequency. Specifically, each infrared LED is controlled using a 555 timer. The 555 timer is configured in astable mode to generate a PWM signal. In this mode, the output frequency is determined by two resistors (R1 and R2) and a capacitor (C1). The frequency formula is: f = 1.44 / ((R_1 + 2R_2) * C_1).

[0160] Step 3: Integrate the control circuit described in step 1 into the PCB board to realize the control of the infrared LED array.

[0161] Step 4: Set the frequency for the lights in the array. Set the frequency of the light with ID 1 to 1kHz. Starting with the light with ID 2, the frequency of each light increases by 0.1kHz. The last light with ID (m*n) has a frequency of 1+(m*n-1)*0.1kHz.

[0162] By following these steps, you can create an ID-coded infrared LED array, where each ID has a unique frequency. This device can be used as a visual calibration board for event camera calibration.

[0163] After that, 3D vision calibration is performed. The specific steps for calibration are as follows:

[0164] Step 1: Install the filter. Attach a 940nm bandpass filter to the camera lens to ensure that only light near the 940nm band passes through. This significantly reduces background and noise interference, ensuring that the majority of the data collected by the camera is valid, reducing the total data volume and laying the foundation for real-time calculations.

[0165] Step 2. Install the camera according to the actual scene to be measured. This method requires two event cameras to form a binocular stereo vision system. When installing, make sure that the field of view of the two cameras covers the area to be measured and there is enough overlapping area. The baseline distance and camera angle are determined by the distance and range of the object to be measured. Among them, the baseline distance is set to 1 / 4 of the distance to the object to be measured. This distance is larger than the baseline distance of a conventional binocular system to obtain higher accuracy. Regarding the camera angle, generally speaking, the larger the camera angle, the closer the measurement distance, the smaller the measurement range, and the higher the measurement accuracy; the smaller the angle, the longer the measurement distance, the larger the measurement range, and the lower the measurement accuracy. The angle is determined according to the actual situation of the object to be measured. After the installation is completed, the camera cannot be moved or the angle adjusted during the calibration process and the measurement process.

[0166] Step 3: Perform intrinsic calibration on both event cameras. This method involves moving a frequency-encoded infrared LED calibration plate within the camera's field of view. Based on the event stream data captured by the camera, the plate's position in the image is calculated in real time. Finally, the camera's intrinsic parameter matrix and distortion parameters are calculated using Zhang's calibration method.

[0167] The specific steps for calculating the calibration plate position and calibration through the event stream are as follows:

[0168] (1) Create a new matrix representing the state of each pixel, called the state matrix S. The shape and size of the state matrix are equal to the camera resolution, where each element corresponds to a pixel at the corresponding position in the camera. The state matrix is a three-channel matrix, and each pixel stores three pieces of information: the timestamp of the last on event; the timestamp of the last off event; and the current state, 1 or 0 representing on or off.

[0169] (2) Create a valid event set E. Whenever a new event occurs, compare the polarity of the new event with the current state of the pixel position in the state matrix according to the state matrix. If the state changes, calculate the frequency f_xy = 1 / (t_2-t_1). xy Indicates the frequency of the event location, t2 indicates the timestamp of the new event, and t1 indicates the timestamp of the last event of this polarity. If the frequency satisfies 0.9 (kHz) <f xy <1+0.1m*n(kHz), the event is added to the valid event set V as a valid event. The attributes that need to be saved for each event in V are pixel position x, y, frequency f, and timestamp t. Subsequently, regardless of whether the state has changed, the element at the corresponding pixel position in the state matrix is updated. The current state in the matrix is replaced by the polarity of the new event, and the timestamp of the new event replaces the timestamp of the previous event of this polarity in the matrix.

[0170] (3) Set E stores the most recent valid events. Events with timestamps older than a threshold are discarded to ensure real-time performance. The time threshold can be set as needed, typically between 1ms and 15ms. Therefore, E is updated in real time based on the event stream.

[0171] (4) Cluster all events in V at the latest moment. You can choose methods such as k-means, DBSCAN, and hierarchical clustering to cluster based on the x, y, and f of the event. After clustering, discard clusters with less than 5 elements to obtain several valid clusters. If the number of valid clusters is (m*n), and the average frequency of events in each cluster corresponds to the preset frequency one by one, and the frequency error does not exceed 0.02kHz, then the calibration plate is determined to be successfully identified. In order of frequency, calculate the pixel coordinate centroid of each cluster as the key point coordinate, obtain the pixel coordinates of each infrared LED, and record them by number.

[0172] (5) After the calibration plate is successfully recognized for the first time, the coordinates of the center point of the calibration plate are recorded and stored in set C. Each subsequent successful recognition needs to be compared with the coordinates of all the center points recorded in C: the Euclidean distance between the newly recognized center point coordinates and each coordinate in C is calculated. If any distance is less than the threshold, the recognition result is discarded. If the distance is greater than or equal to the threshold, the recognition is considered valid, the key point position is recorded, and the center point coordinates are stored in set C. This operation avoids multiple recognitions of similar positions.

[0173] (6) Move the calibration plate in the camera's field of view, identify its position in real time, and continuously accumulate valid recognitions. When the coordinates of the effectively recognized center points in set C evenly cover the entire image and the number of valid recognitions exceeds 20, the shooting is terminated.

[0174] (7) Using the key point position data of the calibration plate in all the above effective recognitions, perform intrinsic calibration according to Zhang’s calibration method to calculate the intrinsic parameter matrix and distortion parameters of the camera.

[0175] Step 3: External parameter calibration. Obtain the rotation matrix R and the normalized translation vector t through the corresponding relationship between the point pairs, and then obtain the true translation vector t through scale calibration. true The specific implementation is as follows:

[0176] (1) Use a single stroboscopic infrared LED as a feature point and move it in the overlapping fields of view of two cameras.

[0177] (2) The two cameras shoot in real time, and the feature points are detected and located according to the recognition method described in step 2. The coordinates of the feature points captured by the two cameras at the same time are paired and saved.

[0178] (3) Move the feature points to collect more point pairs. The feature points should appear as evenly as possible in all positions of the field of view, and the number of point pairs should be greater than 200.

[0179] (4) Solve the essential matrix E based on the corresponding point pairs and the intrinsic parameter matrices of the two cameras.

[0180] (5) Perform singular value decomposition (SVD) on the essential matrix E, and the decomposition formula is E = U∑V T . Where U and V are orthogonal matrices, ∑ is the singular value matrix The rotation vector may be UWV T or UW T V T , the translation vector is the third column vector of U, and its sign may be positive or negative. By triangulating, we determine whether the depth of the point reprojected to the camera is positive, retain a set of solutions with positive depth, and obtain the rotation matrix R and translation vector t.

[0181] (6) The t obtained at this time is a normalized translation vector, which only has direction but no scale information. In order to determine the scale information, scale calibration is required. The specific method is: use a stroboscopic infrared LED calibration board, and the distance d between the key points on the calibration board is known. ij , the three-dimensional coordinates of all points on the calibration plate are obtained by triangulation and the distance between them is calculated Set the loss function to represent the error between the restored distance and the true distance Where s is the scale factor. Use the least squares method or other optimization methods to find s that minimizes the loss function, then the translation vector t with scale true =s·t. The size of the calibration plate used in this step can be selected according to the actual scene. The size of the calibration plate should be as large as possible without exceeding the field of view.

[0182] Then, perform 3D object positioning. Use a stroboscopic infrared LED as a marker to assist in positioning. The specific method is as follows:

[0183] Step 1. Based on the size and shape of the object being measured, select several stroboscopic infrared LEDs as markers and place them on the surface. Assign each LED a unique ID, starting with ID 1. Set the frequency of the LED with ID 1 to 1kHz. Starting with ID 2, increase the frequency by 0.1kHz for each LED. The last LED, ID N, has a frequency of 1 + (N-1) * 0.1kHz.

[0184] Step 2: Position the infrared LEDs according to the method described in step 2 of the calibration section to obtain the pixel coordinates of each LED in the two cameras.

[0185] Step 3: Perform distortion correction and binocular stereo correction on the pixel coordinates based on the camera's internal and external parameters and distortion parameters.

[0186] Step 4: According to the detected frequencies, match the key points detected in the left and right cameras to obtain N groups of point pairs.

[0187] Step 5: Determine the depth Z using triangulation. The formula is Z = (f·B) / d, where f is the focal length, B is the baseline distance, and d is the pixel difference between corresponding points in the two cameras. Finally, convert the pixel coordinates to 3D coordinates.

[0188] Step 6: Based on the three-dimensional coordinates of all points, the position and surface morphology of the object being measured can be obtained, thus achieving 3D positioning.

[0189] Figure 2 FIG1 shows a flow chart of a three-dimensional positioning algorithm based on infrared stroboscopic features of an event camera provided by an embodiment of the present application. Figure 2 As shown, the method includes steps S210 to S280.

[0190] In one embodiment, the infrared stroboscopic feature 3D positioning algorithm based on the event camera is implemented as follows:

[0191] S210: New event input: Receive the real-time light change event stream captured by the event camera as the algorithm input source.

[0192] S220: Status change judgment: By comparing the current event with the status matrix record value, if the difference exceeds a preset threshold (such as ΔI>10%), it is determined to be a status change event.

[0193] S230: State matrix update: directly update the corresponding pixel state value for events that do not reach the threshold, and keep spatial continuity records.

[0194] S240: Calculation of strobe period: For a state mutation event, the flicker frequency of the infrared LED is calculated by using the timestamp difference (f=1 / Δt).

[0195] S250: Valid event screening: Filter interference events according to a preset strobe frequency range (eg, 8-12 Hz) and retain valid events that meet the LED characteristics of the calibration board.

[0196] S260: Spatiotemporal clustering analysis: Use the density-based DBSCAN algorithm to cluster valid events based on spatial proximity and temporal continuity.

[0197] S270: Frequency feature matching: Compare the cluster center frequency with the calibration board LED frequency coding table to determine the corresponding feature point ID.

[0198] S280: 3D centroid calculation: Combined with the binocular camera parameters, the 3D centroid coordinates of each feature point cluster are calculated using the least squares method.

[0199] Based on the same concept, the embodiment of the present application provides a three-dimensional visual calibration and positioning system based on an event camera. Figure 3 The three-dimensional visual calibration and positioning system based on an event camera provided in an embodiment of the present application is described in detail.

[0200] Figure 3 This is a structural block diagram of a three-dimensional visual calibration and positioning system based on an event camera shown in an embodiment of the present application.

[0201] like Figure 3 As shown, the event camera-based 3D visual calibration and positioning system may include:

[0202] A construction module 310 is configured to construct a light source array, the light source array comprising a plurality of independent light sources assigned unique frequency codes, and to construct a binocular vision system using at least two event cameras, the event cameras being equipped with filters that match the wavelength of light signals from the light source array;

[0203] An extraction module 320 is configured to collect a light signal event stream from a light source array through a binocular vision system and extract a valid event set from the light signal event stream, wherein the valid event set includes a pixel position, a light signal frequency, and a timestamp;

[0204] The construction module 310 is further configured to identify the pixel coordinate centroid of each independent light source by performing cluster analysis on the light signal frequency and pixel position, and establish a mapping relationship between the frequency code and the pixel coordinate centroid;

[0205] an arrangement module 330 for arranging a plurality of auxiliary light sources on the surface of the target object, wherein the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array;

[0206] The extraction module 320 is further configured to collect a target light signal event stream of the target object through the binocular vision system, extract a target valid event set from the target light signal event stream, and determine a pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set based on a mapping relationship;

[0207] A calculation module 340 is used to correct the pixel coordinate centroid set and perform binocular matching based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculate the three-dimensional spatial coordinates of the auxiliary light source by triangulation;

[0208] The calculation module 340 is also used to calculate the position of the target object through a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

[0209] In one embodiment, the extraction module 320 is specifically used to create a state matrix, the dimension of the state matrix is consistent with the resolution of the event camera, and each element of the state matrix records the last event timestamp and current polarity state of the corresponding pixel; the polarity of the new event in the light signal event stream is compared with the current polarity state of the corresponding pixel in the state matrix. When the polarity state changes, the frequency value is calculated based on the difference between the timestamp of the new event and the timestamp of the last event of the corresponding pixel recorded in the state matrix; valid events whose frequency values are within the preset frequency coding range of the independent light source are screened, and the pixel position, frequency and timestamp of the valid event are stored in the valid event set.

[0210] In one embodiment, the construction module 310 is specifically used to cluster the events in the valid event set based on the light signal frequency and pixel position to generate multiple event clusters; remove event clusters whose number of elements is lower than a preset number threshold in the multiple event clusters, and retain the valid event clusters; calculate the pixel coordinate centroid of the valid event cluster, and verify the matching degree between the average frequency of the valid event cluster and the preset frequency coding of the independent light source; establish a mapping relationship between the centroid coordinates of the valid event cluster whose matching degree is within the preset error range and the frequency coding.

[0211] In one embodiment, the construction module 310 is also used to correct and binocularly match the pixel coordinate centroid set based on the preset event camera's intrinsic parameter matrix and distortion parameters, and the extrinsic parameter matrix of the binocular vision system, and before calculating the three-dimensional spatial coordinates of the auxiliary light source through triangulation, move the light source array at multiple angles within the field of view of each event camera, collect the light signal event stream of the light source array in real time, and extract valid recognition results; extract multiple sets of calibration plate key point position data based on the mapping relationship between the frequency code and the pixel coordinate centroid in the valid recognition results; calculate the intrinsic parameter matrix and distortion parameters of the event camera through a plane calibration algorithm based on the multiple sets of calibration plate key point position data; when the number of valid recognition results reaches a preset number threshold and the calibration plate key point position data covers the complete field of view of the event camera, output the intrinsic parameter matrix and distortion parameters of the event camera.

[0212] In one embodiment, the construction module 310 is also used to move a single frequency-coded light source within the overlapping field of view of the binocular vision system, and synchronously collect light signal event stream data from two event cameras; match the pixel coordinates of the same light source detected by the two event cameras according to the frequency coding to generate multiple sets of corresponding point pairs; based on the corresponding point pairs and the intrinsic parameter matrix, calculate the rotation matrix and translation vector through essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system.

[0213] In one embodiment, the calculation module 340 is specifically used to use the camera focal length, principal point coordinate parameters and distortion parameters in the intrinsic parameter matrix to perform nonlinear distortion correction on the pixel coordinate centroid set to generate corrected pixel coordinates; based on the rotation matrix and translation vector in the extrinsic parameter matrix, the coordinate system of the event camera of the binocular vision system is stereo epipolar corrected to complete the coplanarity and row alignment of the imaging planes of the event camera, forming a binocular stereo correction mapping relationship; according to the binocular stereo correction mapping relationship, the corrected pixel coordinates are subjected to epipolar constraint matching to obtain the target pixel coordinates, and the horizontal disparity value is calculated by determining the matching point pairs of the same auxiliary light source in the images of different event cameras; according to the focal length parameters in the intrinsic parameter matrix, the baseline distance parameters and disparity values in the extrinsic parameter matrix, the depth information is calculated by the triangulation formula, and based on the depth information and the target pixel, the three-dimensional space coordinates are converted.

[0214] In one embodiment, the calculation module 340 is specifically used to match the three-dimensional spatial coordinates of the auxiliary light source with the distribution information of the auxiliary light source to perform feature point matching to establish an initial posture correspondence; construct an objective function based on an iterative nearest point algorithm, and optimize the rotation matrix and translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source; when the iterative optimization meets the preset convergence conditions, the optimized rotation matrix and translation vector are output to obtain the posture of the target object to complete positioning.

[0215] Figure 3 Each module in the system shown has the function of implementing Figure 1 and Figure 2 The functions of each step in the embodiment can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0216] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application is shown.

[0217] The electronic device may include a processor 410 and a memory 420 storing computer program instructions.

[0218] Specifically, the processor 410 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0219] The memory 420 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 420 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 420 may include removable or non-removable (or fixed) media. Where appropriate, the memory 420 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 420 is a non-volatile solid-state memory.

[0220] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present disclosure.

[0221] The processor 410 reads and executes computer program instructions stored in the memory 420 to implement any one of the event camera-based three-dimensional visual calibration and positioning methods in the above embodiments.

[0222] In one example, the electronic device may further include a communication interface 430 and a bus 440. Figure 4 As shown, the processor 410 , the memory 420 , and the communication interface 430 are connected via a bus 440 and communicate with each other.

[0223] The communication interface 430 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0224] Bus 440 includes hardware, software or both, and the parts of online data flow metering equipment are coupled to each other. For example, but not limitation, bus can include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 440 can include one or more buses. Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0225] The electronic device can execute the three-dimensional visual calibration and positioning method based on the event camera in the embodiment of the present application, thereby realizing the combination of Figure 1 and Figure 2 Described is an event-based 3D visual calibration and localization method.

[0226] In addition, in conjunction with the event camera-based 3D visual calibration and positioning methods in the above embodiments, embodiments of the present application may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement any of the event camera-based 3D visual calibration and positioning methods in the above embodiments.

[0227] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0228] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0229] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0230] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0231] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A three-dimensional visual calibration and positioning method based on an event camera, characterized in that: include: Constructing a light source array, the light source array comprising a plurality of independent light sources assigned unique frequency codes, and constructing a binocular vision system using at least two event cameras, the event cameras being equipped with filters that match the wavelength of light signals from the light source array; Collecting a light signal event stream of the light source array through the binocular vision system, and extracting a valid event set from the light signal event stream, wherein the valid event set includes a pixel position, a light signal frequency, and a timestamp; By performing cluster analysis on the light signal frequency and the pixel position, the pixel coordinate centroid of each independent light source is identified, and a mapping relationship between the frequency code and the pixel coordinate centroid is established; Arrange a plurality of auxiliary light sources on the surface of the target object, wherein the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array; collecting a target light signal event stream of the target object through the binocular vision system, extracting a target valid event set from the target light signal event stream, and determining a pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set according to the mapping relationship; Based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, the pixel coordinate centroid set is corrected and binocular matching is performed, and the three-dimensional spatial coordinates of the auxiliary light source are calculated by triangulation; According to the three-dimensional spatial coordinates and the distribution information of the auxiliary light source, the position of the target object is calculated by a rigid transformation algorithm to complete positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

2. The method according to claim 1, characterized in that The extracting a valid event set from the optical signal event stream comprises: Creating a state matrix, wherein the dimension of the state matrix is consistent with the resolution of the event camera, and each element of the state matrix records the last event timestamp and current polarity state of the corresponding pixel; Comparing the polarity of a new event in the light signal event stream with the current polarity state of the corresponding pixel in the state matrix, and when the polarity state changes, calculating a frequency value based on the difference between the timestamp of the new event and the timestamp of the previous event of the corresponding pixel recorded in the state matrix; Valid events whose frequency values are within a preset frequency coding range of the independent light source are screened, and pixel positions, frequencies, and timestamps of the valid events are stored in the valid event set.

3. The method according to claim 1, characterized in that The method of performing cluster analysis on the optical signal frequency and the pixel position to identify the pixel coordinate centroid of each independent light source and establishing a mapping relationship between the frequency code and the pixel coordinate centroid includes: Clustering the events in the valid event set based on the light signal frequency and the pixel position to generate a plurality of event clusters; removing event clusters whose number of elements is lower than a preset threshold value from the plurality of event clusters, and retaining valid event clusters; Calculating the pixel coordinate centroid of the valid event cluster and verifying the matching degree between the average frequency of the valid event cluster and the preset frequency encoding of the independent light source; A mapping relationship is established between the centroid coordinates of the valid event cluster whose matching degree meets the preset error range and the frequency code.

4. The method according to claim 1, wherein Before correcting the pixel coordinate centroid set and performing binocular matching based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculating the three-dimensional spatial coordinates of the auxiliary light source by triangulation, the method includes: The light source array is moved at multiple angles within the field of view of each event camera to collect the light signal event stream of the light source array in real time and extract effective recognition results; Extracting multiple sets of calibration plate key point position data according to the mapping relationship between the frequency code and the pixel coordinate centroid in the effective recognition result; Based on the multiple sets of calibration plate key point position data, calculating the intrinsic parameter matrix and distortion parameters of the event camera through a plane calibration algorithm; When the number of valid recognition results reaches a preset number threshold and the key point position data of the calibration plate covers the complete field of view of the event camera, the intrinsic parameter matrix and distortion parameters of the event camera are output.

5. The method according to claim 4, characterized in that The method further comprises: Moving a single frequency-coded light source within the overlapping fields of view of the binocular vision system to synchronously collect light signal event stream data from the two event cameras; Matching pixel coordinates of the same light source detected by the two event cameras according to the frequency coding to generate multiple sets of corresponding point pairs; Based on the corresponding point pairs and the intrinsic parameter matrix, the rotation matrix and the translation vector are calculated by essential matrix decomposition to obtain the extrinsic parameter matrix of the binocular vision system.

6. The method according to claim 1, wherein The method of correcting and binocular matching the pixel coordinate centroid set based on the intrinsic parameter matrix, the distortion parameter, and the extrinsic parameter matrix, and calculating the three-dimensional space coordinates of the auxiliary light source by triangulation includes: Using the camera focal length, principal point coordinate parameters and the distortion parameters in the intrinsic parameter matrix, nonlinear distortion correction is performed on the pixel coordinate centroid set to generate corrected pixel coordinates; Based on the rotation matrix and translation vector in the extrinsic parameter matrix, the coordinate system of the event camera of the binocular vision system is subjected to stereo epipolar correction to ensure that the imaging planes of the event cameras are coplanar and aligned, thereby forming a binocular stereo correction mapping relationship; According to the binocular stereo correction mapping relationship, epipolar constraint matching is performed on the corrected pixel coordinates to obtain target pixel coordinates, and horizontal disparity values are calculated by determining matching point pairs of the same auxiliary light source in images of different event cameras; Depth information is calculated using a triangulation formula according to the focal length parameter in the intrinsic parameter matrix, the baseline distance parameter in the extrinsic parameter matrix, and the disparity value, and the three-dimensional space coordinates are converted based on the depth information and the target pixel.

7. The method according to claim 1, characterized in that The method of calculating the position of the target object by a rigid transformation algorithm based on the three-dimensional space coordinates and the distribution information of the auxiliary light source to complete positioning includes: Perform feature point matching on the three-dimensional spatial coordinates of the auxiliary light source and the distribution information of the auxiliary light source to establish an initial posture correspondence relationship; Constructing an objective function based on an iterative closest point algorithm, optimizing a rotation matrix and a translation vector by minimizing the spatial distance residual between the three-dimensional spatial coordinates of the auxiliary light source and the corresponding coordinates in the distribution information of the auxiliary light source; When the iterative optimization satisfies the preset convergence conditions, the optimized rotation matrix and translation vector are output to obtain the position and posture of the target object to complete the positioning.

8. A three-dimensional visual calibration and positioning system based on an event camera, characterized in that: The system comprises: A construction module is used to construct a light source array, wherein the light source array includes multiple independent light sources assigned with unique frequency codes, and a binocular vision system is constructed by at least two event cameras, wherein the event cameras are equipped with filters that match the wavelength of the light signals of the light source array; an extraction module, configured to collect a light signal event stream of the light source array through the binocular vision system, and extract a valid event set from the light signal event stream, wherein the valid event set includes a pixel position, a light signal frequency, and a timestamp; The construction module is further configured to identify the pixel coordinate centroid of each of the independent light sources by performing cluster analysis on the light signal frequency and the pixel position, and establish a mapping relationship between the frequency code and the pixel coordinate centroid; an arrangement module, configured to arrange a plurality of auxiliary light sources on the surface of the target object, wherein the frequency coding of the auxiliary light sources is the same as the frequency coding rule of the light source array; The extraction module is further configured to collect a target light signal event stream of the target object through the binocular vision system, extract a target valid event set from the target light signal event stream, and determine a pixel coordinate centroid set corresponding to the frequency encoding of each auxiliary light source in the target valid event set according to the mapping relationship; A calculation module is used to correct and perform binocular matching on the pixel coordinate centroid set based on the preset intrinsic parameter matrix and distortion parameters of the event camera and the extrinsic parameter matrix of the binocular vision system, and calculate the three-dimensional spatial coordinates of the auxiliary light source by triangulation; The calculation module is also used to calculate the position of the target object through a rigid transformation algorithm based on the three-dimensional spatial coordinates and the distribution information of the auxiliary light source to complete positioning. The distribution information of the auxiliary light source includes the geometric topological relationship and relative position information of the auxiliary light source on the surface of the target object.

9. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the three-dimensional visual calibration and positioning method based on the event camera as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the three-dimensional visual calibration and positioning method based on an event camera according to any one of claims 1 to 7 is implemented.

Images