Lidar-based odometry method, apparatus, and electronic device

Abstract

The application discloses a kind of odometer method, device and electronic equipment based on laser radar, the method includes: obtaining first point cloud data collected by laser radar at current time and second point cloud data collected at last time;Determine the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data;Based on the preset convolutional neural network model, the first image and the second image are processed to obtain target feature vector;The target feature vector is pulse coded based on the preset pulse neural network model, and the pulse sequence obtained by forward propagation pulse coding is outputted, and pulse weighting value is outputted;Based on the pulse weighting value, the translation increment and the rotation increment between the current time and the last time are obtained.The scheme, by preset pulse neural network model, point cloud data is pulse coded and analyzed, and the calculation amount is small and energy consumption is low.

Description

Technical Field

[0001] This invention relates to the field of positioning and navigation technology, and in particular to a method, apparatus and electronic device for odometer based on lidar. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is one of the key core technologies of mobile autonomous systems, especially indispensable for odometry. In recent years, Artificial Neural Networks (ANNs) and Deep Neural Networks (DNNs) have played an increasingly important role in mobile autonomous systems, such as autonomous vehicles, drones, and collaborative robots, with SLAM solutions based on neural networks becoming increasingly common. However, current odometry technologies based on ANNs and DNNs have high computational and energy costs. Therefore, reducing the computational and energy costs of neural networks is an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a method, apparatus, and electronic device for odometer based on lidar.

[0004] In a first aspect, embodiments of the present invention provide an odometer method based on lidar, comprising:

[0005] Acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment;

[0006] Determine the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data;

[0007] Based on a preset convolutional neural network model, the first image and the second image are processed to obtain the target feature vector;

[0008] The target feature vector is pulse-coded based on a preset spiking neural network model, and the pulse weighted value is output through the pulse sequence obtained by forward propagation pulse coding.

[0009] Based on the pulse weighting value, the translation increment and rotation increment between the current time and the previous time are obtained.

[0010] Optionally, after acquiring the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment, the method further includes:

[0011] Outlier removal is performed on the first point cloud data and the second point cloud data respectively to obtain the first point cloud data and the second point cloud data after outlier removal.

[0012] Normalization is performed on the first point cloud data after outlier removal and the second point cloud data after outlier removal, respectively, to obtain the normalized first point cloud data and the normalized second point cloud data.

[0013] Determining the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data includes:

[0014] The first image is determined based on the first point cloud data after normalization, and the second image is determined based on the second point cloud data after normalization.

[0015] Optionally, the first point cloud data and the second point cloud data are data in a Cartesian coordinate system, and determining the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data includes:

[0016] Transform the first point cloud data and the second point cloud data from the Cartesian coordinate system to the spherical coordinate system;

[0017] In the spherical coordinate system, the first point cloud data is mapped onto a four-channel 2D image to obtain the first image, and the second point cloud data is mapped onto a four-channel 2D image to obtain the second image. For each pixel in the image, the four channels include the three-dimensional coordinates corresponding to the pixel and the reflection intensity corresponding to the pixel.

[0018] Optionally, the step of processing the first image and the second image based on a preset convolutional neural network model to obtain the target feature vector includes:

[0019] The first image and the second image are combined to obtain a combined eight-channel image;

[0020] Based on the preset convolutional neural network model, the eight-channel image is processed to obtain the target feature vector.

[0021] Optionally, the step of processing the first image and the second image based on a preset convolutional neural network model to obtain the target feature vector includes:

[0022] Based on the preset convolutional neural network model, the first image and the second image are processed to output an initial feature vector;

[0023] The initial feature vector is normalized to obtain the target feature vector.

[0024] Optionally, the step of pulse coding the target feature vector based on a preset spiking neural network model, and outputting a pulse weighted value through the pulse sequence obtained by forward propagation pulse coding, includes:

[0025] The target feature vector is processed T times based on the preset spiking neural network model to obtain the pulse weighted value output in each cycle, resulting in a total of T sets of pulse weighted values, where T is a positive integer.

[0026] The processing steps for each loop are as follows: pulse encoding is performed on the target feature vector, the pulse sequence obtained by pulse encoding is propagated forward, the membrane voltage of each layer is calculated, the pulse weighted value obtained in this loop is output, and the membrane voltage of each layer calculated in this loop is used for the next loop processing; after T loops, the state of the spiking neuron is reset.

[0027] Optionally, the preset spiking neural network model includes a first sub-model and a second sub-model. The first sub-model is used to output a first pulse weighted value related to the translation increment, and the second sub-model is used to output a second pulse weighted value related to the rotation increment. Both the first and second sub-models contain three layers of spiking neurons, and the first layer is a pulse coding layer. All layers are fully connected. The method further includes:

[0028] In each iteration, the target feature vector is used as the current magnitude of a DC current source. The pulse coding layers in the first and second sub-models are updated with membrane potentials using the following formulas to obtain the pulse sequence for this iteration:

[0029]

[0030] The pulse sequence processed in this loop is propagated forward. The second and third layers in the first and second sub-models are updated with membrane potentials using the following formulas, and the pulse weighted values ​​obtained in this loop are output:

[0031]

[0032] Where, τ m R is the membrane time constant. m V is the membrane resistance, V is the membrane voltage, and I is the sum of the synaptic currents generated by the firing behavior of each presynaptic neuron. rest It is the resting potential.

[0033] Optionally, the preset spiking neural network is constructed through the following steps:

[0034] An initial neural network model is constructed based on a preset network structure, wherein the neurons in the initial neural network model are artificial neural network neurons;

[0035] The initial neural network model is trained based on the first sample set to obtain the trained first model parameters;

[0036] The artificial neural network neurons are replaced with spiking neural network neurons to form an initial spiking neural network model, wherein the initial model parameters of the initial spiking neural network model are the first model parameters;

[0037] The initial spiking neural network model is trained based on the second sample set to obtain a trained spiking neural network model, which is then used as the preset spiking neural network model.

[0038] Secondly, embodiments of the present invention provide an odometer based on lidar, comprising:

[0039] The data acquisition module is used to acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment;

[0040] The image conversion module is used to determine a first image corresponding to the first point cloud data and a second image corresponding to the second point cloud data;

[0041] The first processing module is used to process the first image and the second image based on a preset convolutional neural network model to obtain a target feature vector;

[0042] The second processing module is used to perform pulse coding on the target feature vector based on a preset spiking neural network model, and output pulse weighting values ​​through the pulse sequence obtained by forward propagation pulse coding.

[0043] The third processing module is used to obtain the translation increment and rotation increment between the current time and the previous time based on the pulse weighting value.

[0044] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the operation instructions corresponding to the above-described lidar-based odometer method.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps corresponding to the lidar-based odometer method provided in the first aspect.

[0046] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects:

[0047] The scheme in this embodiment acquires first point cloud data collected by the lidar at the current moment and second point cloud data collected at the previous moment; determines a first image corresponding to the first point cloud data and a second image corresponding to the second point cloud data; processes the first and second images based on a preset convolutional neural network model to obtain a target feature vector; performs pulse encoding on the target feature vector based on a preset spiking neural network model, and outputs a pulse weighted value through the pulse sequence obtained by forward propagation pulse encoding; and obtains the translation increment and rotation increment between the current moment and the previous moment based on the pulse weighted value. In this scheme, pulse encoding of point cloud data is achieved through a preset spiking neural network model, and odometry analysis is performed through the preset spiking neural network. Since the spiking neural network uses a mathematical model that is closer to biology, it has greater biological interpretability. At the same time, due to the sparsity of information expression in the spiking neural network itself, the computation of neuromorphic hardware is very sparse, resulting in lower energy consumption. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a lidar-based odometer method provided in this specification.

[0050] Figure 2 A flowchart illustrating a lidar-based odometer method provided in the embodiments of this specification;

[0051] Figure 3 A schematic diagram of a lidar-based odometer device provided for an embodiment of this specification;

[0052] Figure 4 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Detailed Implementation

[0053] This application provides a method, apparatus, electronic device, and medium for odometer based on lidar.

[0054] The overall technical solution of this application embodiment is as follows: acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment; determine the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data; process the first image and the second image based on a preset convolutional neural network model to obtain the target feature vector; perform pulse encoding on the target feature vector based on a preset spiking neural network model, and output the pulse weighting value through the pulse sequence obtained by forward propagation pulse encoding; based on the pulse weighting value, obtain the translation increment and rotation increment between the current moment and the previous moment.

[0055] In this scheme, point cloud data is pulse encoded by a preset spiking neural network model, and odometry analysis is performed by the preset spiking neural network. Since the spiking neural network uses a mathematical model that is closer to that of biology, it is more biologically interpretable. At the same time, due to the sparsity of information expression of the spiking neural network itself, the computation of neuromorphic hardware is very sparse and the energy consumption is lower.

[0056] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0057] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0058] like Figure 1 The diagram shown is a flowchart of a lidar-based odometer method provided in an embodiment of this specification. The method includes the following steps:

[0059] Step S101: Acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment;

[0060] Step S102: Determine the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data;

[0061] Step S103: Based on a preset convolutional neural network model, process the first image and the second image to obtain the target feature vector;

[0062] Step S104: Perform pulse coding on the target feature vector based on a preset spiking neural network model, and output the pulse weighted value through the pulse sequence obtained by forward propagation pulse coding;

[0063] Step S105: Based on the pulse weighting value, obtain the translation increment and rotation increment between the current time and the previous time.

[0064] The methods provided in the embodiments of this specification can be applied to terminal devices or servers that perform positioning and mapping, and can also be applied to systems composed of terminal devices and servers; no limitation is made here. Taking a terminal device as an example, the terminal device can be a robot, vehicle, or other device capable of positioning and mapping.

[0065] For ease of understanding, a robot is used as an example to illustrate the method provided in the embodiments of this specification. The robot is equipped with at least one lidar, which can scan the environment in which the robot is located and obtain point cloud data.

[0066] In step S101, the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment are acquired. Specifically, the lidar is in a continuous scanning state and will continuously acquire point cloud data. For each frame of point cloud data acquired, it can be analyzed and processed based on the point cloud data of that frame and the adjacent previous frame of point cloud data according to the method provided in the embodiments of this specification.

[0067] To avoid the point cloud data acquisition device (e.g., a robot) affecting data acquisition, in this embodiment, outlier removal can be performed on the first and second point cloud data respectively, resulting in outlier-removed first and second point cloud data. In this embodiment, data with distance values ​​less than a threshold in the point cloud data can be removed as outlier data. The threshold can be set according to actual needs; for example, it can be 0.3m, 0.4m, etc., and is not limited here.

[0068] It should be noted that in the embodiments of this specification, after each frame of point cloud data is collected, outlier removal can be performed on that frame of point cloud data. Therefore, after obtaining the first point cloud data at the current moment, only outlier processing needs to be performed on the first point cloud data. For the second point cloud data at the previous moment, the result of outlier processing on the second point cloud data at the previous moment can be directly read.

[0069] To ensure that two consecutive frames of point cloud data have the same numerical scale and avoid instability in the neural network caused by excessive differences in values ​​between different features, the first and second point cloud data need to be normalized. In practice, the first and second point cloud data (after outlier removal) can be normalized separately to obtain normalized first and second point cloud data. The normalization formula can be as follows:

[0070]

[0071]

[0072] Among them, NP′ k For the first point cloud data after normalization, NP′ k-1 For the normalized second point cloud data, Np k,i For the i-th point in the first point cloud data after normalization, Np k-1,j For the j-th point in the normalized second point cloud data, p k,i ∈P′ k p k-1,j ∈P′ k-1 , P′ k For the first point cloud data after outlier removal, P′ k-1 This is the second point cloud data after outlier removal. In the point cloud data, each point p = (p... x ,p y ,p z All are 3D points, N and M are P'' points respectively. k and P′ k-1 The number of midpoint clouds.

[0073] In step S102, after determining the first point cloud data and the second point cloud data, the first point cloud data and the second point cloud data can be converted into images for subsequent analysis through image processing. In the embodiments of this specification, after obtaining the normalized first point cloud data and the normalized second point cloud data, the first image can be determined based on the normalized first point cloud data, and the second image can be determined based on the normalized second point cloud data.

[0074] Specifically, the method of converting point cloud data into an image can be selected according to actual needs. For example, point cloud data can be converted into a 2D planar image, and the specific parameters of the image can also be set according to actual needs.

[0075] In the specific implementation process, the first point cloud data and the second point cloud data can both be data in a rectangular coordinate system. The image conversion can be achieved through the following steps: converting the first point cloud data and the second point cloud data from the rectangular coordinate system to the spherical coordinate system; in the spherical coordinate system, mapping the first point cloud data to a four-channel 2D image to obtain the first image, and mapping the second point cloud data to a four-channel 2D image to obtain the second image, wherein, for each pixel in the image, the four channels include the three-dimensional coordinates corresponding to the pixel and the reflection intensity corresponding to the pixel.

[0076] It should be noted that converting point cloud data into images allows for the use of mature image processing techniques while reducing computational and memory consumption. Continuing with the example above, we can convert NP′... k and NP′ k-1 Mapped to a 4-channel 2D image, the first image Img is obtained. k Second image Img k-1 Image conversion can be performed by mapping 3D point cloud data to a four-channel 2D image using the following formula (3).

[0077]

[0078] Where (u,v) T These are the coordinates of pixels in a 2D image. Each pixel has four channels: its 3D coordinates (x, y, z) in space, and its reflection intensity. U represents the horizontal axis of the image, V represents the vertical axis, and the origin is located at the top left corner of the image. The depth d = ((p x ) 2 +(p y ) 2 ) 1 / 2 f h and f v These are the horizontal field of view (FOV) and the vertical field of view, with the vertical field of view f... v =f vu +f vl Where f vu The part above the horizontal plane, f vl It is the part below the horizontal level, δ h and δ v These are the horizontal and vertical resolutions, represented by pixels.

[0079] As can be seen from the above formula, the coordinates in the rectangular coordinate system can be transformed to the spherical coordinate system through trigonometric function transformation, and then further mapped to the 2D image.

[0080] After obtaining the first image and the second image, image processing can be performed on the first image and the second image through step S103. In specific implementation, step S103 can be implemented through the following steps: integrating the first image and the second image to obtain an integrated eight-channel image; processing the eight-channel image based on the preset convolutional neural network model to obtain the target feature vector.

[0081] It should be noted that the structure of the preset convolutional neural network model can be set according to actual needs. For example, the number of layers, the number of neurons in each layer, and the connection method between layers can all be set according to actual needs.

[0082] Using the example of the four-channel 2D image above, after obtaining two consecutive 2D images Img... k-1 1mg k Afterwards, you can follow the instructions for Img. k-1 1mg k The images are sequentially integrated into an eight-channel image and input into a preset neural network model. In some embodiments, a network similar to ResNet-18 can be used as the preset convolutional neural network model, and the specific network structure and parameters are shown in Table 1.

[0083] Table 1

[0084]

[0085] Where H is the height of the image and W is the width of the image; s and p are the convolution stride and image padding size, respectively. s(1,2) means that the convolution stride is 1 in the height direction and 2 in the width direction, and p(1,0) is similar. maxpool represents max pooling, average pool represents average pooling, and 1000-d means that the dimension of this network layer is 1×1000.

[0086] In some embodiments, the output of a preset convolutional neural network model can be used as the target feature vector. In other embodiments, the output of the preset convolutional neural network model can be further processed to obtain the target feature vector.

[0087] In the specific implementation process, the target feature vector can be obtained in the following way: based on the preset convolutional neural network model, the first image and the second image are processed to output an initial feature vector; the initial feature vector is normalized to obtain the target feature vector.

[0088] Specifically, taking the preset convolutional neural network model corresponding to Table 1 above as an example, the initial feature vector RN is output after processing by this model. out Initial feature vector RN out The dimension is 1×1000, and then, for RN out Normalization is performed to obtain the normalized target feature vector NRN. out Normalization can be achieved using the following formula:

[0089]

[0090] Among them, NRN out,i For the target feature vector NRN out The i-th feature in, min(RN) out ) for RN out The minimum value in, max(RN) out ) for RN out The maximum value in.

[0091] It should be noted that the reason for normalization is based on the characteristics of spiking neural networks (SNNs), which utilize pulses to transmit information and fire pulses when the membrane potential exceeds a threshold. If the normalization process is directly applied to the RN... out As a direct current source, information with values ​​less than 0 cannot be transmitted, resulting in information loss and affecting the accuracy of network inference.

[0092] In step S104, the target feature vector is processed by a preset spiking neural network model. In this embodiment, the preset neural network model can perform pulse encoding on the target feature vector and output pulse weighted values. The specific structure of the preset neural network model can be selected according to actual needs and is not limited here.

[0093] In the specific implementation process, the data processing of the preset spiking neural network model can be achieved through the following steps: Based on the preset spiking neural network model, the target feature vector is processed T times in a loop to obtain the pulse weighted value output in each loop, resulting in a total of T sets of pulse weighted values, where T is a positive integer. Each loop processing step is as follows: pulse encoding is performed on the target feature vector; the pulse sequence obtained from the pulse encoding is propagated forward; the membrane voltage of each layer is calculated; the pulse weighted value obtained in this loop is output; and the membrane voltage of each layer calculated in this loop is used for the next loop processing. After the T loops are completed, the state of the spiking neurons is reset.

[0094] Specifically, the target feature vector is the NRN obtained after the above normalization process. out For example, NRN out A preset spiking neural network model is connected in a fully connected manner. In this embodiment of the specification, the preset spiking neural network includes a first sub-model and a second sub-model. The first sub-model is used to output a first pulse weighted value related to the translation increment, and the second sub-model is used to output a second pulse weighted value related to the rotation increment. Both the first sub-model and the second sub-model contain three layers of spiking neurons, and the first layer is a pulse coding layer. All layers are fully connected.

[0095] In each iteration, the target feature vector is used as the current magnitude of a DC current source. The pulse coding layers in the first and second sub-models are updated with membrane potentials using the following formulas to obtain the pulse sequence for this iteration:

[0096]

[0097] It should be noted that the coding neurons in the pulse coding layer can adopt the Leaky-Integrate-and-Fire (LIF) model, and the corresponding membrane potential update equation is Equation (5). In Equation (5), τ m R is the membrane time constant. m V is the membrane resistance, V is the membrane voltage, and I is the sum of the synaptic currents generated by the firing behavior of each presynaptic neuron. rest It is the resting potential.

[0098] Among them, R m V rest The membrane potential threshold V of the emitted pulse th The value of R can be set according to actual needs. In some embodiments, R m The value is 1 ohm, V rest The value is 1 volt, V th It is 0.54 volts. A pulse sequence can be obtained through processing with a pulse coding layer.

[0099] The pulse sequence processed in this loop is propagated forward. The second and third layers in the first and second sub-models can be updated with membrane potentials using formula (6), and the pulse weighted values ​​obtained in this loop are output:

[0100]

[0101] It should be noted that the second and third layers of the preset spiking neural network model can adopt the integrated-and-fire (IF) model, and the membrane potential update equation is Equation (6), where R m The membrane potential threshold V of the emitted pulse th The value of R can be set according to actual needs. In some embodiments, R m The value is 1 ohm, V th It is 1 volt.

[0102] Based on the above steps, the target feature vector is processed in T loops. Each loop outputs the pulse weighted value obtained in this loop. The value of T can be set according to actual needs. Taking T as 10 as an example, the above process is executed 10 times. In each loop, formulas (5) and (6) use the calculation results of the membrane voltage in the previous loop. After the T loops are completed, the state of the spiking neuron is reset.

[0103] In the embodiments of this specification, if T is 10, then the pulse weighting value output in each cycle is:

[0104] (t x,i ,t y,i ,t z,i ,r x,i ,r y,i ,r z,i ,r w,i ), i = 1, 2, ..., 10

[0105] Among them, t x,i t y,i t z,i The pulse weighting value r, which corresponds to the output of the i-th iteration of the first sub-model and is used to calculate the translation increment, is... x,i r y,i r z,i r w,i The pulse weighting value used to calculate the rotation increment is output in the i-th iteration of the second sub-model.

[0106] In step S105, after obtaining the pulse weighting value, the translation increment and rotation increment can be further calculated. Continuing with the example above, the translation increment Δt... k-1→k = (Δx, Δy, Δz) and rotation increment Δr k-1→k = (Δx, Δy, Δz, Δw) can be calculated using the following formula:

[0107]

[0108]

[0109]

[0110]

[0111] It should be noted that when normalizing using formulas (1) and (2), the scales of the first point cloud data and the second point cloud data are unified, reducing or eliminating the differences between the values. In the numerical description of translation, formula (8) is needed to restore this difference.

[0112] In the embodiments of this specification, the preset spiking neural network model can be constructed through the following steps: constructing an initial neural network model based on a preset network structure, wherein the neurons in the initial neural network model are artificial neural network neurons; training the initial neural network model based on a first sample set to obtain trained first model parameters; replacing the artificial neural network neurons with spiking neural network neurons to form an initial spiking neural network model, wherein the initial model parameters of the initial spiking neural network model are the first model parameters; and training the initial spiking neural network model based on a second sample set to obtain a trained spiking neural network model, which serves as the preset spiking neural network model.

[0113] Specifically, the preset network structure can be set according to actual needs. For example, the preset network structure can include a first sub-network and a second sub-network, which correspond to the first sub-model and the second sub-model mentioned above, respectively. Each sub-network is a three-layer structure. The first sub-network is a fully connected network of 256-128-(64-1, 64-1, 64-1), and the second sub-network is a fully connected network of 256-128-(64-1, 64-1, 64-1, 64-1).

[0114] It should be noted that since SNNs transmit information in the form of pulses, the membrane potential needs to reach a certain threshold to trigger the pulse. During model training, if the initial parameters of the model are not set reasonably, the threshold for triggering the pulse may not be reached. In this case, even if the error has been backpropagated, the pulse may still not be triggered, resulting in unstable model updates.

[0115] To avoid the aforementioned problems, in the embodiments of this specification, ANN neurons can be used instead of spiking neurons for model training to obtain the first model parameters. Specifically, when constructing the initial neural network model, the neurons in the initial neural network model are set as artificial neural network neurons, i.e., ANN neurons. The initial neural network model is trained using a first sample set. The training method can be self-supervised, or other training methods can also be used. When the number of iterations of model training reaches a threshold or the error between the model output and the correct odometer is within a preset range, the model training can be considered complete, and the first model parameters are obtained.

[0116] Furthermore, the ANN neurons in the trained model are replaced with spiking neurons to form the initial spiking neural network model to be trained. This model is then trained using a second set of samples until the network parameters converge, resulting in the final trained spiking neural network model, which serves as the preset spiking neural network model. The first and second samples can be the same or different; this is not limited here.

[0117] To better illustrate the lidar-based odometer method provided in the embodiments of this specification, please refer to... Figure 2 Here is a flowchart of the odometry method based on lidar. Figure 2 In the preset spiking neural network model, there are a first sub-model and a second sub-model. The first sub-model is a fully connected network with a 256-128-(64-1, 64-1, 64-1) byte structure, and the second sub-model is also a fully connected network with a 256-128-(64-1, 64-1, 64-1, 64-1) byte structure. Figure 2 The corresponding processing steps are as follows:

[0118] Step 1: Obtain the point cloud data P at the current moment. k And the point cloud data P from the previous moment k-1 k refers to the current time, and k-1 refers to the previous time.

[0119] Step 2: For P respectively k and P k-1 Perform outlier removal to obtain P′ after outlier removal. k and P′ k-1 ;

[0120] Step 3: Apply the above formulas (1) and (2) to P′ respectively. k and P′ k-1 After normalization, we obtain NP′ k and NP′ k-1 ;

[0121] Step 4: Use formula (3) to process the 3D point cloud data NP′ respectively. k and NP′ k-1 Mapped to a four-channel 2D image, resulting in Img. k and Img k-1 ;

[0122] Step 5: Press Img to merge two consecutive 2D images. k 1mg k-1 The images are sequentially integrated into an 8-channel image and input into a pre-defined convolutional neural network model to obtain a vector RN with a dimension of 1×1000. out ;

[0123] Step 6: Apply formula (4) to RN out Normalization is performed to obtain NRN out and NRN out As a DC current source, it is connected to the first layer of a pre-defined spiking neural network model in a fully connected manner;

[0124] Step 7: Use the first layer of the preset spiking neural network model as the spiking coding layer, and update the membrane potential of the spiking coding neurons using formula (5);

[0125] Among them, NRN out The magnitude of the current, acting as a direct current source, is input to the pulse coding neurons of the pulse coding layer, resulting in a pulse sequence s with a dimension of 1×256. LIF ;

[0126] Step 8: Convert the pulse sequence s LIF Forward propagation, full connectivity between layers, output layer outputs pulse weighted value, the second and third layers of the preset spiking neural network model are updated with the membrane potential using formula (6);

[0127] In this embodiment, the time window T, i.e., the number of loops, is 10. Therefore, steps 7 and 8 need to be looped 10 times. In each loop, steps 7 and 8 use the calculated membrane voltage results from the previous loop. After the loop ends, the neuron membrane voltage is reset. Figure 2 The t in the formula corresponds to Δt in formula (8). k-1→k The integral result, r, corresponds to Δr in formula (10). k-1→k The integral result T is used here to describe the rotation using quaternions;

[0128] Step 9: Calculate the translation increment from k-1 to k using formulas (7) and (8), and calculate the rotation increment from k-1 to k using formulas (9) and (10).

[0129] It should be noted that, in the embodiments of this specification, the preferred spiking neural network model is a fully connected network of 256-128-(64-1, 64-1, 64-1) and 256-128-(64-1, 64-1, 64-1, 64-1). In the specific design process, if network structures of 256-256-256-3 and 256-256-256-4 are used, there will be a large number of connections and spiking points, resulting in high communication pressure. To optimize this problem, the number of neurons is reduced, and the spiking points are sparsified. The aforementioned fully connected networks of 256-128-(64-1, 64-1, 64-1) and 256-128-(64-1, 64-1, 64-1, 64-1) can effectively solve the above problems. This model structure has fewer connections, lower computational load, and faster inference speed.

[0130] In summary, the solutions provided in the embodiments of this specification have at least the following technical effects:

[0131] 1) The pulse coding proposed in the embodiments of this specification is an improved frequency coding scheme. First, by taking advantage of the leakage characteristics of the LIF model, the output pulse sequence is sparser than other frequency coding methods. Second, since the membrane potential update is nonlinear, the information expression ability is stronger. Third, because the spiking neural network uses a mathematical model that is closer to that of biology, it is more biologically interpretable.

[0132] 2) The fully connected networks in the embodiments of this specification are more lightweight. Specifically, the preset spiking neural networks used in this solution are 256-128-(64-1,64-1,64-1) and 256-128-(64-1,64-1,64-1,64-1), which are used for the translational and rotational increments of the inference robot from time k-1 to k, respectively. Compared to the 256-256-256-3 and 256-256-256-4 fully connected networks, they have fewer connections, thus requiring less computation and resulting in faster inference speed.

[0133] 3) Lower energy consumption. Due to the sparse information representation of SNNs, the computation of neuromorphic hardware is very sparse. At the same time, with even sparser pulse coding and fewer connections, the energy consumption can be further reduced.

[0134] Based on the same inventive concept, embodiments of this specification also provide an odometer device based on lidar, such as... Figure 3 As shown, the device includes:

[0135] The data acquisition module 301 is used to acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment;

[0136] Image conversion module 302 is used to determine a first image corresponding to the first point cloud data and a second image corresponding to the second point cloud data;

[0137] The first processing module 303 is used to process the first image and the second image based on a preset convolutional neural network model to obtain a target feature vector;

[0138] The second processing module 304 is used to perform pulse coding on the target feature vector based on a preset pulse neural network model, and output pulse weighting value through the pulse sequence obtained by forward propagation pulse coding.

[0139] The third processing module 305 is used to obtain the translation increment and rotation increment between the current time and the previous time based on the pulse weighting value.

[0140] Optionally, the device further includes:

[0141] The outlier removal module is used to remove outliers from the first point cloud data and the second point cloud data respectively, so as to obtain the first point cloud data and the second point cloud data after outlier removal.

[0142] The first normalization processing module is used to normalize the first point cloud data after outlier removal and the second point cloud data after outlier removal, respectively, to obtain the normalized first point cloud data and the normalized second point cloud data.

[0143] The image conversion module 302 is used to determine the first image based on the first point cloud data after normalization processing, and to determine the second image based on the second point cloud data after normalization processing.

[0144] Optionally, the image conversion module 302 is used for:

[0145] Transform the first point cloud data and the second point cloud data from the Cartesian coordinate system to the spherical coordinate system;

[0146] In the spherical coordinate system, the first point cloud data is mapped onto a four-channel 2D image to obtain the first image, and the second point cloud data is mapped onto a four-channel 2D image to obtain the second image. For each pixel in the image, the four channels include the three-dimensional coordinates corresponding to the pixel and the reflection intensity corresponding to the pixel.

[0147] Optionally, the first processing module 303 is used for:

[0148] The first image and the second image are combined to obtain a combined eight-channel image;

[0149] Based on the preset convolutional neural network model, the eight-channel image is processed to obtain the target feature vector.

[0150] Optionally, the first processing module 303 is used for:

[0151] Based on the preset convolutional neural network model, the first image and the second image are processed to output an initial feature vector;

[0152] The initial feature vector is normalized to obtain the target feature vector.

[0153] Optionally, the second processing module 304 is used for:

[0154] The target feature vector is processed T times based on the preset spiking neural network model to obtain the pulse weighted value output in each cycle, resulting in a total of T sets of pulse weighted values, where T is a positive integer.

[0155] The processing steps for each loop are as follows: pulse encoding is performed on the target feature vector, the pulse sequence obtained by pulse encoding is propagated forward, the membrane voltage of each layer is calculated, the pulse weighted value obtained in this loop is output, and the membrane voltage of each layer calculated in this loop is used for the next loop processing; after T loops, the state of the spiking neuron is reset.

[0156] Optionally, the preset spiking neural network model includes a first sub-model and a second sub-model. The first sub-model is used to output a first pulse weighted value related to the translation increment, and the second sub-model is used to output a second pulse weighted value related to the rotation increment. Both the first and second sub-models contain three layers of spiking neurons, and the first layer is a pulse coding layer. All layers are fully connected. The fourth processing module is used for:

[0157] In each iteration, the target feature vector is used as the current magnitude of a DC current source. The pulse coding layers in the first and second sub-models are updated with membrane potentials using the following formulas to obtain the pulse sequence for this iteration:

[0158]

[0159] The pulse sequence processed in this loop is propagated forward. The second and third layers in the first and second sub-models are updated with membrane potentials using the following formulas, and the pulse weighted values ​​obtained in this loop are output:

[0160]

[0161] Where, τ m R is the membrane time constant. m V is the membrane resistance, V is the membrane voltage, and I is the sum of the synaptic currents generated by the firing behavior of each presynaptic neuron. rest It is the resting potential.

[0162] Optionally, the preset spiking neural network is constructed through the following steps:

[0163] An initial neural network model is constructed based on a preset network structure, wherein the neurons in the initial neural network model are artificial neural network neurons;

[0164] The initial neural network model is trained based on the first sample set to obtain the trained first model parameters;

[0165] The artificial neural network neurons are replaced with spiking neural network neurons to form an initial spiking neural network model, wherein the initial model parameters of the initial spiking neural network model are the first model parameters;

[0166] The initial spiking neural network model is trained based on the second sample set to obtain a trained spiking neural network model, which is then used as the preset spiking neural network model.

[0167] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the lidar-based odometer method, and will not be elaborated upon here.

[0168] Based on the same inventive concept, embodiments of this specification also provide an electronic device, such as... Figure 4 As shown, it includes a memory 408, a processor 402, and a computer program stored in the memory 408 and executable on the processor 402. When the processor 402 executes the program, it implements the steps of the lidar-based odometer method described above.

[0169] Among them, Figure 4 In this document, a bus architecture (represented by bus 400) is used. Bus 400 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 402 and memory represented by memory 408. Bus 400 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 406 provides an interface between bus 400 and receiver 401 and transmitter 403. Receiver 401 and transmitter 403 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 402 is responsible for managing bus 400 and general processing, while memory 408 can be used to store data used by processor 402 during operation.

[0170] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned lidar-based odometer method.

[0171] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0175] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for odometer measurement based on lidar, characterized in that, include: Acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment; Determine the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data; Based on a preset convolutional neural network model, the first image and the second image are processed to obtain the target feature vector; Pulse coding is performed on the target feature vector based on a preset spiking neural network model, and a pulse weighting value is output through the pulse sequence obtained by forward propagation pulse coding, including: performing pulse coding on the target feature vector based on the preset spiking neural network model. T The process is repeated several times, resulting in the pulse weighted value output from each iteration, thus obtaining a total of [number] pulse weighted values. T Group pulse weighting, T is a positive integer; wherein, the processing steps of each loop are as follows: pulse encoding is performed on the target feature vector, the pulse sequence obtained by pulse encoding is propagated forward, and the membrane voltage of each layer is calculated, the pulse weighted value obtained in this loop is output, and the membrane voltage of each layer calculated in this loop is used for the next loop processing; T After the next loop ends, the state of the spiking neurons is reset; Based on the pulse weighting value, the translation increment and rotation increment between the current time and the previous time are obtained; The preset spiking neural network model includes a first sub-model and a second sub-model. The first sub-model outputs a first pulse weighted value related to the translation increment, and the second sub-model outputs a second pulse weighted value related to the rotation increment. Both the first and second sub-models contain three layers of spiking neurons, with the first layer being a pulse coding layer. All layers are fully connected. In each iteration, the target feature vector is used as the current magnitude of a DC current source. The pulse coding layers in the first and second sub-models update the membrane potential using the following formulas to obtain the pulse sequence for the current iteration: ； The pulse sequence processed in this loop is propagated forward. The second and third layers in the first and second sub-models are updated with membrane potentials using the following formulas, and the pulse weighted values ​​obtained in this loop are output: ； in, It is the membrane time constant. It is a membrane resistor. It is the membrane voltage. It is the sum of the synaptic currents generated by the firing behavior of each presynaptic neuron. It is the resting potential.

2. The method as described in claim 1, characterized in that, After acquiring the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment, the method further includes: Outlier removal is performed on the first point cloud data and the second point cloud data respectively to obtain the first point cloud data and the second point cloud data after outlier removal. Normalization is performed on the first point cloud data after outlier removal and the second point cloud data after outlier removal, respectively, to obtain the normalized first point cloud data and the normalized second point cloud data. Determining the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data includes: The first image is determined based on the first point cloud data after normalization, and the second image is determined based on the second point cloud data after normalization.

3. The method as described in claim 1 or 2, characterized in that, The first point cloud data and the second point cloud data are data in a Cartesian coordinate system. Determining the first image corresponding to the first point cloud data and the second image corresponding to the second point cloud data includes: Transform the first point cloud data and the second point cloud data from the Cartesian coordinate system to the spherical coordinate system; In the spherical coordinate system, the first point cloud data is mapped onto a four-channel 2D image to obtain the first image, and the second point cloud data is mapped onto a four-channel 2D image to obtain the second image. For each pixel in the image, the four channels include the three-dimensional coordinates corresponding to the pixel and the reflection intensity corresponding to the pixel.

4. The method as described in claim 3, characterized in that, The process of processing the first image and the second image based on a preset convolutional neural network model to obtain a target feature vector includes: The first image and the second image are combined to obtain a combined eight-channel image; Based on the preset convolutional neural network model, the eight-channel image is processed to obtain the target feature vector.

5. The method as described in claim 1, characterized in that, The process of processing the first image and the second image based on a preset convolutional neural network model to obtain a target feature vector includes: Based on the preset convolutional neural network model, the first image and the second image are processed to output an initial feature vector; The initial feature vector is normalized to obtain the target feature vector.

6. The method as described in claim 1, characterized in that, The preset spiking neural network is constructed through the following steps: An initial neural network model is constructed based on a preset network structure, wherein the neurons in the initial neural network model are artificial neural network neurons; The initial neural network model is trained based on the first sample set to obtain the trained first model parameters; The artificial neural network neurons are replaced with spiking neural network neurons to form an initial spiking neural network model, wherein the initial model parameters of the initial spiking neural network model are the first model parameters; The initial spiking neural network model is trained based on the second sample set to obtain a trained spiking neural network model, which is then used as the preset spiking neural network model.

7. A odometer device based on lidar, characterized in that, include: The data acquisition module is used to acquire the first point cloud data collected by the lidar at the current moment and the second point cloud data collected at the previous moment; The image conversion module is used to determine a first image corresponding to the first point cloud data and a second image corresponding to the second point cloud data; The first processing module is used to process the first image and the second image based on a preset convolutional neural network model to obtain a target feature vector; The second processing module is used to perform pulse coding on the target feature vector based on a preset spiking neural network model, and output pulse weighting values ​​through the pulse sequence obtained by forward propagation pulse coding, including: performing pulse coding on the target feature vector based on the preset spiking neural network model. T The process is repeated several times, resulting in the pulse weighted value output from each iteration, thus obtaining a total of [number] pulse weighted values. T Group pulse weighting, T is a positive integer; wherein, the processing steps of each loop are as follows: pulse encoding is performed on the target feature vector, the pulse sequence obtained by pulse encoding is propagated forward, and the membrane voltage of each layer is calculated, the pulse weighted value obtained in this loop is output, and the membrane voltage of each layer calculated in this loop is used for the next loop processing; T After the next loop ends, the state of the spiking neurons is reset; The third processing module is used to obtain the translation increment and rotation increment between the current time and the previous time based on the pulse weighting value; The preset spiking neural network model includes a first sub-model and a second sub-model. The first sub-model outputs a first pulse weighted value related to the translation increment, and the second sub-model outputs a second pulse weighted value related to the rotation increment. Both the first and second sub-models contain three layers of spiking neurons, with the first layer being a pulse coding layer. All layers are fully connected. In each iteration, the target feature vector is used as the current magnitude of a DC current source. The pulse coding layers in the first and second sub-models update the membrane potential using the following formulas to obtain the pulse sequence for the current iteration: ； The pulse sequence processed in this loop is propagated forward. The second and third layers in the first and second sub-models are updated with membrane potentials using the following formulas, and the pulse weighted values ​​obtained in this loop are output: ； in, It is the membrane time constant. It is a membrane resistor. It is the membrane voltage. It is the sum of the synaptic currents generated by the firing behavior of each presynaptic neuron. It is the resting potential.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the method steps as described in any one of claims 1-6.

Images